Started by user Jenkins Admin Obtained pipelines/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test.groovy from git https://github.com/PingCAP-QE/ci.git Loading library tipipeline@main Library tipipeline@main is cached. Copying from home. [Pipeline] Start of Pipeline [Pipeline] readJSON [Pipeline] readTrusted Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-28p2x Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-28p2x is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-wfr2t-2vbtq --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "5a01d75eb95d3f572f9305c4b6fd0a19600c420a" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-wfr2t" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-28p2x" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-28p2x" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-28p2x" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-28p2x in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout The recommended git tool is: git No credentials specified Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 > git rev-list --no-walk e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] timeout Timeout set to expire in 1 hr 0 min [Pipeline] { [Pipeline] stage [Pipeline] { (Debug info) [Pipeline] sh + printenv PROW_JOB_ID=c548ef96-eb30-4461-bb19-187e41d862e3 JENKINS_NODE_COOKIE=c6a48aa5-76e8-4c65-87cb-75f0cbdfa872 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/ GOLANG_VERSION=1.21.10 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-28p2x HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Debug info BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1796069089731416066","prowjobid":"c548ef96-eb30-4461-bb19-187e41d862e3","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct _=/usr/bin/printenv POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#590 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089731416066 GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=tests SHLVL=3 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-wfr2t GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-d4c91f37a2cd54d8a1ba68aedcc7e1b173b616003bd6c892b8b1a5e40c8eb9f7 NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-wfr2t ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-28p2x GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-28p2x GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=590 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz + echo ------------------------- ------------------------- + go env GO111MODULE='' GOARCH='amd64' GOBIN='' GOCACHE='/home/jenkins/.cache/go-build' GOENV='/home/jenkins/.config/go/env' GOEXE='' GOEXPERIMENT='' GOFLAGS='' GOHOSTARCH='amd64' GOHOSTOS='linux' GOINSECURE='' GOMODCACHE='/go/pkg/mod' GONOPROXY='' GONOSUMDB='' GOOS='linux' GOPATH='/go' GOPRIVATE='' GOPROXY='http://goproxy.apps.svc,https://proxy.golang.org,direct' GOROOT='/usr/local/go' GOSUMDB='sum.golang.org' GOTMPDIR='' GOTOOLCHAIN='auto' GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64' GOVCS='' GOVERSION='go1.21.10' GCCGO='gccgo' GOAMD64='v1' AR='ar' CC='gcc' CXX='g++' CGO_ENABLED='1' GOMOD='/dev/null' GOWORK='' CGO_CFLAGS='-O2 -g' CGO_CPPFLAGS='' CGO_CXXFLAGS='-O2 -g' CGO_FFLAGS='-O2 -g' CGO_LDFLAGS='-O2 -g' PKG_CONFIG='pkg-config' GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build4176068600=/tmp/go-build -gno-record-gcc-switches' + echo ------------------------- ------------------------- + echo 'debug command: kubectl -n jenkins-tiflow exec -ti ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-28p2x bash' debug command: kubectl -n jenkins-tiflow exec -ti ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-28p2x bash [Pipeline] container [Pipeline] { [Pipeline] sh + dig github.com ; <<>> DiG 9.18.16 <<>> github.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8196 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ; COOKIE: add79d1d9519a543 (echoed) ;; QUESTION SECTION: ;github.com. IN A ;; ANSWER SECTION: github.com. 22 IN A 20.205.243.166 ;; Query time: 1 msec ;; SERVER: 169.254.25.10#53(169.254.25.10) (UDP) ;; WHEN: Thu May 30 06:40:50 UTC 2024 ;; MSG SIZE rcvd: 77 [Pipeline] script [Pipeline] { [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Check diff files) [Pipeline] container [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $token [Pipeline] { [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/11200/files?page=1&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/11200/files?page=1&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/11200/files?page=2&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/11200/files?page=2&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] } [Pipeline] // withCredentials [Pipeline] echo pr_diff_files: [cdc/kv/shared_client.go, go.mod, go.sum, pkg/spanz/hash_map.go, pkg/util/seahash/LICENSE.md, pkg/util/seahash/seahash.go, pkg/util/seahash/seahash_test.go, scripts/check-copyright.sh] [Pipeline] echo diff file not matched: cdc/kv/shared_client.go [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Checkout) [Pipeline] timeout Timeout set to expire in 10 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] cache Cache restored successfully (git/pingcap/tiflow/rev-26b901c-645f6fc) 208132608 bytes in 0.83 secs (249620226 bytes/sec) [Pipeline] { [Pipeline] retry [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] sh git version 2.36.6 Reinitialized existing Git repository in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/.git/ .git HEAD is now at 645f6fc14 This is an automated cherry-pick of #11193 POST git-upload-pack (686 bytes) POST git-upload-pack (973 bytes) From https://github.com/pingcap/tiflow = [up to date] release-7.5 -> origin/release-7.5 645f6fc14..8f92142d7 refs/pull/11200/head -> origin/pr/11200/head Previous HEAD position was 645f6fc14 This is an automated cherry-pick of #11193 HEAD is now at 26b901cb5 dm: upgrade go-mysql library (#11043) (#11189) ๐Ÿšง Checkouting to base SHA:26b901cb5b23f8c42b05e5a00514b52bef2efe24... HEAD is now at 26b901cb5 dm: upgrade go-mysql library (#11043) (#11189) โœ… Checked. ๐ŸŽ‰ ๐Ÿงพ HEAD info: 26b901cb5b23f8c42b05e5a00514b52bef2efe24 26b901cb5 dm: upgrade go-mysql library (#11043) (#11189) bf93d7d63 dm: fix update dml loss if binary column is primary key (#10685) (#10691) 39a17af6e dep(*): move diff package to the tiflow and get rid of the tidb-tools (#11164) (#11168) ๐Ÿšง Pre-merge heads of pull requests to base SHA: 26b901cb5b23f8c42b05e5a00514b52bef2efe24 ... Updating 26b901cb5..8f92142d7 Fast-forward cdc/kv/shared_client.go | 2 +- go.mod | 10 --- go.sum | 34 --------- pkg/spanz/hash_map.go | 2 +- pkg/util/seahash/LICENSE.md | 9 +++ pkg/util/seahash/seahash.go | 159 +++++++++++++++++++++++++++++++++++++++ pkg/util/seahash/seahash_test.go | 75 ++++++++++++++++++ scripts/check-copyright.sh | 2 +- 8 files changed, 246 insertions(+), 47 deletions(-) create mode 100644 pkg/util/seahash/LICENSE.md create mode 100644 pkg/util/seahash/seahash.go create mode 100644 pkg/util/seahash/seahash_test.go ๐Ÿงพ Pre-merged result: 8f92142d7874bfad2a8db6ca100604c383fca9df 8f92142d7 fix 645f6fc14 This is an automated cherry-pick of #11193 26b901cb5 dm: upgrade go-mysql library (#11043) (#11189) โœ… Pre merged ๐ŸŽ‰ โœ… ~~~~~All done.~~~~~~ [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // retry [Pipeline] } Cache not saved (git/pingcap/tiflow/rev-26b901c-8f92142 already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (prepare) [Pipeline] timeout Timeout set to expire in 20 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/third_party_download [Pipeline] { [Pipeline] retry [Pipeline] { [Pipeline] sh + cd ../tiflow + ./scripts/download-integration-test-binaries.sh release-7.5 Download binaries... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 41 100 41 0 0 1522 0 --:--:-- --:--:-- --:--:-- 1576 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 41 100 41 0 0 959 0 --:--:-- --:--:-- --:--:-- 976 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 41 100 41 0 0 1145 0 --:--:-- --:--:-- --:--:-- 1171 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 41 100 41 0 0 984 0 --:--:-- --:--:-- --:--:-- 1000 >>> download tidb-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tidb/39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef/centos7/tidb-server.tar.gz 2024-05-30 14:41:27 URL:http://fileserver.pingcap.net/download/builds/pingcap/tidb/39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef/centos7/tidb-server.tar.gz [512424861/512424861] -> "tmp/tidb-server.tar.gz" [1] >>> download pd-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/pd/634e05a87ee0459846e48e8acea3f6bc5f6973a7/centos7/pd-server.tar.gz 2024-05-30 14:41:46 URL:http://fileserver.pingcap.net/download/builds/pingcap/pd/634e05a87ee0459846e48e8acea3f6bc5f6973a7/centos7/pd-server.tar.gz [173743994/173743994] -> "tmp/pd-server.tar.gz" [1] >>> download tikv-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tikv/1a2c88dcc22c62e1f138d440956214ce367e838b/centos7/tikv-server.tar.gz 2024-05-30 14:42:31 URL:http://fileserver.pingcap.net/download/builds/pingcap/tikv/1a2c88dcc22c62e1f138d440956214ce367e838b/centos7/tikv-server.tar.gz [817702020/817702020] -> "tmp/tikv-server.tar.gz" [1] >>> download tiflash.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tiflash/release-7.5/db0862135d4a94904069a37ec50c69cfe6099952/centos7/tiflash.tar.gz 2024-05-30 14:42:55 URL:http://fileserver.pingcap.net/download/builds/pingcap/tiflash/release-7.5/db0862135d4a94904069a37ec50c69cfe6099952/centos7/tiflash.tar.gz [263371625/263371625] -> "tmp/tiflash.tar.gz" [1] >>> download minio.tar.gz from http://fileserver.pingcap.net/download/minio.tar.gz 2024-05-30 14:42:59 URL:http://fileserver.pingcap.net/download/minio.tar.gz [17718777/17718777] -> "tmp/minio.tar.gz" [1] >>> download go-ycsb from http://fileserver.pingcap.net/download/builds/pingcap/go-ycsb/test-br/go-ycsb 2024-05-30 14:43:00 URL:http://fileserver.pingcap.net/download/builds/pingcap/go-ycsb/test-br/go-ycsb [45975512/45975512] -> "third_bin/go-ycsb" [1] >>> download jq from http://fileserver.pingcap.net/download/builds/pingcap/test/jq-1.6/jq-linux64 2024-05-30 14:43:02 URL:http://fileserver.pingcap.net/download/builds/pingcap/test/jq-1.6/jq-linux64 [3953824/3953824] -> "third_bin/jq" [1] >>> download etcd.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/etcd-v3.4.7-linux-amd64.tar.gz 2024-05-30 14:43:02 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/etcd-v3.4.7-linux-amd64.tar.gz [17310840/17310840] -> "tmp/etcd.tar.gz" [1] >>> download sync_diff_inspector.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/sync_diff_inspector_hash-79f1fd1e_linux-amd64.tar.gz 2024-05-30 14:43:04 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/sync_diff_inspector_hash-79f1fd1e_linux-amd64.tar.gz [55896256/55896256] -> "tmp/sync_diff_inspector.tar.gz" [1] >>> download schema-registry.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/schema-registry.tar.gz 2024-05-30 14:43:13 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/schema-registry.tar.gz [278386006/278386006] -> "tmp/schema-registry.tar.gz" [1] Download SUCCESS + ls -alh ./bin total 1.6G drwxr-sr-x 6 jenkins jenkins 4.0K May 30 14:43 . drwxr-sr-x 19 jenkins jenkins 4.0K May 30 14:43 .. drwxr-sr-x 2 jenkins jenkins 4.0K May 19 2023 bin drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 etc -rwxr-xr-x 1 jenkins jenkins 17M Apr 2 2020 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M May 30 14:43 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M May 30 14:43 jq drwxr-sr-x 3 jenkins jenkins 4.0K May 10 2023 lib lrwxrwxrwx 1 jenkins jenkins 13 May 27 16:52 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1016K Nov 7 2023 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 May 27 16:52 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 358K Nov 7 2023 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 May 27 16:52 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 May 27 16:52 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.6M May 27 16:18 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 88M May 27 16:44 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M Jul 29 2020 minio -rwxr-xr-x 1 jenkins jenkins 33M May 28 20:27 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 43M May 28 20:27 pd-ctl -rwxr-xr-x 1 jenkins jenkins 30M May 28 20:27 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 29M May 28 20:27 pd-recover -rwxr-xr-x 1 jenkins jenkins 103M May 28 20:26 pd-server -rwxr-xr-x 1 jenkins jenkins 25M May 28 20:27 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 29M May 28 20:27 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 share -rwxr-xr-x 1 jenkins jenkins 29M May 28 20:27 stores-dump -rwxr-xr-x 1 jenkins jenkins 133M May 6 16:40 sync_diff_inspector -rwxr-xr-x 1 jenkins jenkins 199M May 29 23:11 tidb-server -rwxr-xr-x 1 jenkins jenkins 377M May 27 16:52 tiflash -rwxr-xr-x 1 jenkins jenkins 357M May 28 18:52 tikv-server + make check_third_party_binary /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/bin/tidb-server /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/bin/tikv-server /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/bin/pd-server /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/bin/tiflash /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/bin/pd-ctl /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/bin/sync_diff_inspector /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/bin/go-ycsb /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/bin/etcdctl /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/bin/jq /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/bin/minio /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/bin/bin/schema-registry-start + cd - /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/third_party_download + mkdir -p bin + mv ../tiflow/bin/bin ../tiflow/bin/etc ../tiflow/bin/etcdctl ../tiflow/bin/go-ycsb ../tiflow/bin/jq ../tiflow/bin/lib ../tiflow/bin/libc++.so.1 ../tiflow/bin/libc++.so.1.0 ../tiflow/bin/libc++abi.so.1 ../tiflow/bin/libc++abi.so.1.0 ../tiflow/bin/libgmssl.so ../tiflow/bin/libgmssl.so.3 ../tiflow/bin/libgmssl.so.3.0 ../tiflow/bin/libtiflash_proxy.so ../tiflow/bin/minio ../tiflow/bin/pd-api-bench ../tiflow/bin/pd-ctl ../tiflow/bin/pd-heartbeat-bench ../tiflow/bin/pd-recover ../tiflow/bin/pd-server ../tiflow/bin/pd-tso-bench ../tiflow/bin/regions-dump ../tiflow/bin/share ../tiflow/bin/stores-dump ../tiflow/bin/sync_diff_inspector ../tiflow/bin/tidb-server ../tiflow/bin/tiflash ../tiflow/bin/tikv-server ./bin/ + ls -alh ./bin total 1.6G drwxr-sr-x 6 jenkins jenkins 4.0K May 30 14:43 . drwxr-sr-x 3 jenkins jenkins 4.0K May 30 14:43 .. drwxr-sr-x 2 jenkins jenkins 4.0K May 19 2023 bin drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 etc -rwxr-xr-x 1 jenkins jenkins 17M Apr 2 2020 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M May 30 14:43 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M May 30 14:43 jq drwxr-sr-x 3 jenkins jenkins 4.0K May 10 2023 lib lrwxrwxrwx 1 jenkins jenkins 13 May 27 16:52 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1016K Nov 7 2023 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 May 27 16:52 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 358K Nov 7 2023 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 May 27 16:52 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 May 27 16:52 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.6M May 27 16:18 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 88M May 27 16:44 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M Jul 29 2020 minio -rwxr-xr-x 1 jenkins jenkins 33M May 28 20:27 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 43M May 28 20:27 pd-ctl -rwxr-xr-x 1 jenkins jenkins 30M May 28 20:27 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 29M May 28 20:27 pd-recover -rwxr-xr-x 1 jenkins jenkins 103M May 28 20:26 pd-server -rwxr-xr-x 1 jenkins jenkins 25M May 28 20:27 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 29M May 28 20:27 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 share -rwxr-xr-x 1 jenkins jenkins 29M May 28 20:27 stores-dump -rwxr-xr-x 1 jenkins jenkins 133M May 6 16:40 sync_diff_inspector -rwxr-xr-x 1 jenkins jenkins 199M May 29 23:11 tidb-server -rwxr-xr-x 1 jenkins jenkins 377M May 27 16:52 tiflash -rwxr-xr-x 1 jenkins jenkins 357M May 28 18:52 tikv-server + ./bin/tidb-server -V Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore + ./bin/pd-server -V Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 + ./bin/tikv-server -V TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + ./bin/tiflash --version TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure + ./bin/sync_diff_inspector --version App Name: sync_diff_inspector v2.0 Release Version: v7.1.5-2-geae7b5d Git Commit Hash: eae7b5d149e30a02a04ec6116e08b0a8f07fe4aa Git Branch: disable-cgo-for-sync-diff UTC Build Time: 2024-05-06 08:40:40 Go Version: go1.22.1 [Pipeline] } [Pipeline] // retry [Pipeline] } [Pipeline] // dir [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] cache Cache not restored (no such key found) [Pipeline] { [Pipeline] sh + ls -alh ./bin total 8.0K drwxr-sr-x 2 jenkins jenkins 4.0K May 30 14:43 . drwxr-sr-x 19 jenkins jenkins 4.0K May 30 14:43 .. + '[' -f ./bin/cdc ']' + make cdc CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-58-g8f92142d7" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-30 06:43:16" -X "github.com/pingcap/tiflow/pkg/version.GitHash=8f92142d7874bfad2a8db6ca100604c383fca9df" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.10 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-58-g8f92142d7"' -o bin/cdc ./cmd/cdc/main.go go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20240527072219-a97e6464c01d go: downloading github.com/pingcap/failpoint v0.0.0-20240412033321-fd0796e60f86 go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/spf13/cobra v1.8.0 go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/fatih/color v1.16.0 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading google.golang.org/grpc v1.64.0 go: downloading github.com/pingcap/kvproto v0.0.0-20240522024016-df42997c2c57 go: downloading github.com/tikv/pd/client v0.0.0-20240210135946-3488a653ddd9 go: downloading github.com/IBM/sarama v1.41.2 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/xdg/scram v1.0.5 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/gin-gonic/gin v1.9.1 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/prometheus/client_golang v1.19.1 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20240510070134-e7f530c64790 go: downloading golang.org/x/net v0.25.0 go: downloading golang.org/x/sync v0.7.0 go: downloading github.com/BurntSushi/toml v1.3.2 go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20240527072219-a97e6464c01d go: downloading github.com/cockroachdb/pebble v0.0.0-20220415182917-06c9d3be25b3 go: downloading go.uber.org/atomic v1.11.0 go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/google/uuid v1.6.0 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading go.etcd.io/etcd/server/v3 v3.5.12 go: downloading golang.org/x/time v0.5.0 go: downloading github.com/swaggo/files v0.0.0-20210815190702-a29dd2bc99b2 go: downloading github.com/swaggo/gin-swagger v1.2.0 go: downloading github.com/tinylib/msgp v1.1.6 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.1 go: downloading github.com/KimMachineGun/automemlimit v0.2.4 go: downloading github.com/aws/aws-sdk-go v1.44.259 go: downloading github.com/shirou/gopsutil/v3 v3.24.4 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/soheilhy/cmux v0.1.5 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading github.com/apache/pulsar-client-go v0.11.0 go: downloading github.com/aws/aws-sdk-go-v2 v1.19.1 go: downloading github.com/benbjohnson/clock v1.3.5 go: downloading github.com/stretchr/testify v1.9.0 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading golang.org/x/sys v0.20.0 go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading cloud.google.com/go v0.112.2 go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2 go: downloading github.com/goccy/go-json v0.10.2 go: downloading github.com/r3labs/diff v1.1.0 go: downloading github.com/cenkalti/backoff/v4 v4.2.1 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading github.com/google/btree v1.1.2 go: downloading github.com/uber-go/atomic v1.4.0 go: downloading github.com/DATA-DOG/go-sqlmock v1.5.0 go: downloading github.com/imdario/mergo v0.3.16 go: downloading github.com/swaggo/swag v1.16.3 go: downloading github.com/xdg/stringprep v1.0.3 go: downloading golang.org/x/crypto v0.23.0 go: downloading github.com/golang/mock v1.6.0 go: downloading github.com/containerd/cgroups v1.0.4 go: downloading github.com/gavv/monotime v0.0.0-20190418164738-30dba4353424 go: downloading github.com/klauspost/compress v1.17.8 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/philhofer/fwd v1.1.1 go: downloading github.com/gin-contrib/sse v0.1.0 go: downloading github.com/golang/protobuf v1.5.4 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/prometheus/common v0.53.0 go: downloading github.com/prometheus/procfs v0.15.0 go: downloading google.golang.org/protobuf v1.34.1 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/aws/smithy-go v1.13.5 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading github.com/stretchr/objx v0.5.2 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading github.com/eapache/go-resiliency v1.4.0 go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 go: downloading github.com/eapache/queue v1.1.0 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/jcmturner/gofork v1.7.6 go: downloading github.com/jcmturner/gokrb5/v8 v8.4.4 go: downloading github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 go: downloading github.com/bits-and-blooms/bitset v1.4.0 go: downloading github.com/linkedin/goavro/v2 v2.11.1 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading golang.org/x/oauth2 v0.18.0 go: downloading google.golang.org/api v0.170.0 go: downloading github.com/pierrec/lz4 v2.6.1+incompatible go: downloading go.etcd.io/etcd/pkg/v3 v3.5.12 go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 go: downloading go.etcd.io/bbolt v1.3.9 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.22.0 go: downloading go.opentelemetry.io/otel/sdk v1.22.0 go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/hashicorp/golang-lru v0.5.1 go: downloading github.com/segmentio/kafka-go v0.4.41-0.20230526171612-f057b1d369cd go: downloading github.com/go-playground/validator/v10 v10.14.0 go: downloading github.com/pelletier/go-toml/v2 v2.0.8 go: downloading github.com/ugorji/go/codec v1.2.11 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/go-mysql-org/go-mysql v1.7.1-0.20240507075657-2bd4573edde2 go: downloading github.com/pingcap/check v0.0.0-20211026125417-57bd13f7b5f0 go: downloading github.com/cakturk/go-netstat v0.0.0-20200220111822-e5b49efee7a5 go: downloading golang.org/x/text v0.15.0 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240521202816-d264139d666e go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/KyleBanks/depth v1.2.1 go: downloading github.com/go-openapi/spec v0.21.0 go: downloading github.com/godbus/dbus/v5 v5.0.4 go: downloading github.com/opencontainers/runtime-spec v1.0.2 go: downloading github.com/cilium/ebpf v0.4.0 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.1 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 go: downloading github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/hashicorp/errwrap v1.0.0 go: downloading github.com/jcmturner/dnsutils/v2 v2.0.0 go: downloading github.com/hashicorp/go-uuid v1.0.3 go: downloading github.com/AthenZ/athenz v1.10.39 go: downloading github.com/spaolacci/murmur3 v1.1.0 go: downloading golang.org/x/mod v0.17.0 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.6 go: downloading go.etcd.io/etcd/raft/v3 v3.5.12 go: downloading github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/pingcap/tipb v0.0.0-20240507090649-2bf6bb0cb996 go: downloading github.com/jonboulle/clockwork v0.4.0 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/blacktear23/go-proxyprotocol v1.0.6 go: downloading github.com/gorilla/mux v1.8.0 go: downloading github.com/pingcap/fn v1.0.0 go: downloading github.com/tiancaiamao/appdash v0.0.0-20181126055449-889f96f722a2 go: downloading github.com/sourcegraph/appdash-data v0.0.0-20151005221446-73f23eafcf67 go: downloading github.com/gorilla/websocket v1.5.1 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/gabriel-vasile/mimetype v1.4.2 go: downloading github.com/leodido/go-urn v1.2.4 go: downloading github.com/go-playground/universal-translator v0.18.1 go: downloading github.com/Masterminds/semver v1.5.0 go: downloading github.com/siddontang/go-log v0.0.0-20180807004314-8d05993dda07 go: downloading github.com/siddontang/go v0.0.0-20180604090527-bdc77568d726 go: downloading github.com/shopspring/decimal v1.3.0 go: downloading github.com/jcmturner/rpc/v2 v2.0.3 go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible go: downloading github.com/99designs/keyring v1.2.1 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/go-openapi/jsonpointer v0.21.0 go: downloading github.com/go-openapi/jsonreference v0.21.0 go: downloading github.com/go-openapi/swag v0.23.0 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/xdg-go/scram v1.1.2 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.1 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/danjacques/gofslock v0.0.0-20220131014315-6e321f4509c8 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/go-logr/logr v1.4.1 go: downloading github.com/golang-jwt/jwt/v4 v4.5.0 go: downloading go.etcd.io/etcd/client/v2 v2.305.12 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading go.opentelemetry.io/proto/otlp v1.1.0 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/iancoleman/strcase v0.2.0 go: downloading github.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/jcmturner/aescts/v2 v2.0.0 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/mailru/easyjson v0.7.7 go: downloading github.com/aws/aws-sdk-go-v2/config v1.18.30 go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.13.29 go: downloading github.com/aws/aws-sdk-go-v2/service/glue v1.58.1 go: downloading github.com/jarcoal/httpmock v1.2.0 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/go-playground/locales v0.14.1 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/dvsekhvalnov/jose2go v1.5.0 go: downloading github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c go: downloading github.com/mtibben/percent v0.2.1 go: downloading golang.org/x/term v0.20.0 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/xdg-go/pbkdf2 v1.0.0 go: downloading github.com/xdg-go/stringprep v1.0.4 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/xitongsys/parquet-go v1.6.0 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/pingcap/badger v1.5.1-0.20230103063557-828f39b09b6d go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/vbauerster/mpb/v7 v7.5.3 go: downloading github.com/sasha-s/go-deadlock v0.3.1 go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.6 go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.3.37 go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.12.14 go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.14 go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.20.1 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/mattn/go-runewidth v0.0.15 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.12.0 go: downloading github.com/josharian/intern v1.0.0 go: downloading k8s.io/api v0.28.6 go: downloading github.com/emirpasic/gods v1.18.1 go: downloading github.com/petermattis/goid v0.0.0-20231207134359-e60b3f734c67 go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.36 go: downloading github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/golang-jwt/jwt/v5 v5.2.0 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.30 go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.30 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading k8s.io/apimachinery v0.28.6 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading k8s.io/klog/v2 v2.120.1 go: downloading github.com/google/gofuzz v1.2.0 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b go: downloading github.com/jmespath/go-jmespath v0.4.0 go: downloading github.com/ardielle/ardielle-go v1.5.2 go: downloading github.com/google/s2a-go v0.1.7 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading github.com/felixge/httpsnoop v1.0.4 + '[' -f ./bin/cdc_kafka_consumer ']' + make kafka_consumer CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-58-g8f92142d7" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-30 06:46:06" -X "github.com/pingcap/tiflow/pkg/version.GitHash=8f92142d7874bfad2a8db6ca100604c383fca9df" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.10 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-58-g8f92142d7"' -o bin/cdc_kafka_consumer ./cmd/kafka-consumer/main.go + '[' -f ./bin/cdc_storage_consumer ']' + make storage_consumer CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-58-g8f92142d7" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-30 06:46:13" -X "github.com/pingcap/tiflow/pkg/version.GitHash=8f92142d7874bfad2a8db6ca100604c383fca9df" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.10 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-58-g8f92142d7"' -o bin/cdc_storage_consumer ./cmd/storage-consumer/main.go + '[' -f ./bin/cdc.test ']' + make integration_test_build cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/failpoint-ctl github.com/pingcap/failpoint/failpoint-ctl go: downloading github.com/pingcap/failpoint v0.0.0-20210316064728-7acb0f0a3dfd go: downloading github.com/sergi/go-diff v1.1.0 CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-58-g8f92142d7" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-30 06:46:21" -X "github.com/pingcap/tiflow/pkg/version.GitHash=8f92142d7874bfad2a8db6ca100604c383fca9df" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.10 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-58-g8f92142d7"' -o bin/cdc_storage_consumer ./cmd/storage-consumer/main.go CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-58-g8f92142d7" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-30 06:46:21" -X "github.com/pingcap/tiflow/pkg/version.GitHash=8f92142d7874bfad2a8db6ca100604c383fca9df" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.10 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-58-g8f92142d7"' -o bin/cdc_kafka_consumer ./cmd/kafka-consumer/main.go CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-58-g8f92142d7" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-30 06:46:21" -X "github.com/pingcap/tiflow/pkg/version.GitHash=8f92142d7874bfad2a8db6ca100604c383fca9df" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.10 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-58-g8f92142d7"' -o bin/cdc_pulsar_consumer ./cmd/pulsar-consumer/main.go $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl enable >/dev/null) go: downloading github.com/PingCAP-QE/go-sqlsmith v0.0.0-20231213065948-336e064b488d go: downloading gorm.io/gorm v1.24.5 go: downloading github.com/chzyer/readline v1.5.1 go: downloading github.com/deepmap/oapi-codegen v1.9.0 go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 go: downloading github.com/gogo/gateway v1.1.0 go: downloading github.com/getkin/kin-openapi v0.80.0 go: downloading github.com/DataDog/zstd v1.5.5 go: downloading github.com/syndtr/goleveldb v1.0.1-0.20210305035536-64b5b1c73954 go: downloading github.com/glebarez/sqlite v1.7.0 go: downloading gorm.io/driver/mysql v1.3.3 go: downloading github.com/shurcooL/httpgzip v0.0.0-20190720172056-320755c1c1b0 go: downloading go.uber.org/dig v1.13.0 go: downloading github.com/glebarez/go-sqlite v1.21.2 go: downloading github.com/mattn/go-shellwords v1.0.12 go: downloading github.com/go-ozzo/ozzo-validation/v4 v4.3.0 go: downloading go.uber.org/ratelimit v0.2.0 go: downloading github.com/VividCortex/mysqlerr v1.0.0 go: downloading github.com/edwingeng/deque v0.0.0-20191220032131-8596380dee17 go: downloading go.uber.org/goleak v1.3.0 go: downloading github.com/bradleyjkemp/grpc-tools v0.2.5 go: downloading github.com/integralist/go-findroot v0.0.0-20160518114804-ac90681525dc go: downloading github.com/jmoiron/sqlx v1.3.3 go: downloading upper.io/db.v3 v3.7.1+incompatible go: downloading github.com/ngaut/log v0.0.0-20210830112240-0124ec040aeb go: downloading github.com/andres-erbsen/clock v0.0.0-20160526145045-9e14626cd129 go: downloading modernc.org/libc v1.50.4 go: downloading modernc.org/sqlite v1.29.8 go: downloading github.com/jinzhu/inflection v1.0.0 go: downloading github.com/jinzhu/now v1.1.5 go: downloading github.com/ghodss/yaml v1.0.0 go: downloading github.com/improbable-eng/grpc-web v0.12.0 go: downloading github.com/rs/cors v1.7.0 go: downloading github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f go: downloading modernc.org/memory v1.8.0 go: downloading modernc.org/mathutil v1.6.0 CGO_ENABLED=1 GO111MODULE=on go test -p 3 --race --tags=intest -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-58-g8f92142d7" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-30 06:46:21" -X "github.com/pingcap/tiflow/pkg/version.GitHash=8f92142d7874bfad2a8db6ca100604c383fca9df" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.10 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-58-g8f92142d7"' -c -cover -covermode=atomic \ -coverpkg=github.com/pingcap/tiflow/... \ -o bin/cdc.test github.com/pingcap/tiflow/cmd/cdc \ || { $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null); exit 1; } CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-58-g8f92142d7" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-30 06:46:21" -X "github.com/pingcap/tiflow/pkg/version.GitHash=8f92142d7874bfad2a8db6ca100604c383fca9df" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.10 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-58-g8f92142d7"' -o bin/cdc ./cmd/cdc/main.go \ || { $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null); exit 1; } $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null) + ls -alh ./bin total 1.1G drwxr-sr-x 2 jenkins jenkins 4.0K May 30 14:51 . drwxr-sr-x 19 jenkins jenkins 4.0K May 30 14:43 .. -rwxr-xr-x 1 jenkins jenkins 208M May 30 14:51 cdc -rwxr-xr-x 1 jenkins jenkins 339M May 30 14:50 cdc.test -rwxr-xr-x 1 jenkins jenkins 178M May 30 14:46 cdc_kafka_consumer -rwxr-xr-x 1 jenkins jenkins 177M May 30 14:46 cdc_pulsar_consumer -rwxr-xr-x 1 jenkins jenkins 177M May 30 14:46 cdc_storage_consumer + ./bin/cdc version Release Version: v7.5.1-58-g8f92142d7 Git Commit Hash: 8f92142d7874bfad2a8db6ca100604c383fca9df Git Branch: HEAD UTC Build Time: 2024-05-30 06:46:21 Go Version: go version go1.21.10 linux/amd64 Failpoint Build: true [Pipeline] } Cache saved successfully (binary/pingcap/tiflow/cdc-integration-test/rev-26b901c-8f92142) 1129025024 bytes in 22.77 secs (49589185 bytes/sec) [Pipeline] // cache [Pipeline] cache Cache not restored (no such key found) [Pipeline] { [Pipeline] sh + cp -r ../third_party_download/bin/bin ../third_party_download/bin/etc ../third_party_download/bin/etcdctl ../third_party_download/bin/go-ycsb ../third_party_download/bin/jq ../third_party_download/bin/lib ../third_party_download/bin/libc++.so.1 ../third_party_download/bin/libc++.so.1.0 ../third_party_download/bin/libc++abi.so.1 ../third_party_download/bin/libc++abi.so.1.0 ../third_party_download/bin/libgmssl.so ../third_party_download/bin/libgmssl.so.3 ../third_party_download/bin/libgmssl.so.3.0 ../third_party_download/bin/libtiflash_proxy.so ../third_party_download/bin/minio ../third_party_download/bin/pd-api-bench ../third_party_download/bin/pd-ctl ../third_party_download/bin/pd-heartbeat-bench ../third_party_download/bin/pd-recover ../third_party_download/bin/pd-server ../third_party_download/bin/pd-tso-bench ../third_party_download/bin/regions-dump ../third_party_download/bin/share ../third_party_download/bin/stores-dump ../third_party_download/bin/sync_diff_inspector ../third_party_download/bin/tidb-server ../third_party_download/bin/tiflash ../third_party_download/bin/tikv-server ./bin/ + ls -alh ./bin total 2.7G drwxr-sr-x 6 jenkins jenkins 4.0K May 30 14:51 . drwxr-sr-x 19 jenkins jenkins 4.0K May 30 14:43 .. drwxr-sr-x 2 jenkins jenkins 4.0K May 30 14:51 bin -rwxr-xr-x 1 jenkins jenkins 208M May 30 14:51 cdc -rwxr-xr-x 1 jenkins jenkins 339M May 30 14:50 cdc.test -rwxr-xr-x 1 jenkins jenkins 178M May 30 14:46 cdc_kafka_consumer -rwxr-xr-x 1 jenkins jenkins 177M May 30 14:46 cdc_pulsar_consumer -rwxr-xr-x 1 jenkins jenkins 177M May 30 14:46 cdc_storage_consumer drwxr-sr-x 4 jenkins jenkins 4.0K May 30 14:51 etc -rwxr-xr-x 1 jenkins jenkins 17M May 30 14:51 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M May 30 14:51 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M May 30 14:51 jq drwxr-sr-x 3 jenkins jenkins 4.0K May 30 14:51 lib lrwxrwxrwx 1 jenkins jenkins 13 May 30 14:51 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1016K May 30 14:51 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 May 30 14:51 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 358K May 30 14:51 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 May 30 14:51 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 May 30 14:51 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.6M May 30 14:51 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 88M May 30 14:51 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M May 30 14:51 minio -rwxr-xr-x 1 jenkins jenkins 33M May 30 14:51 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 43M May 30 14:51 pd-ctl -rwxr-xr-x 1 jenkins jenkins 30M May 30 14:51 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 29M May 30 14:51 pd-recover -rwxr-xr-x 1 jenkins jenkins 103M May 30 14:51 pd-server -rwxr-xr-x 1 jenkins jenkins 25M May 30 14:51 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 29M May 30 14:51 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K May 30 14:51 share -rwxr-xr-x 1 jenkins jenkins 29M May 30 14:51 stores-dump -rwxr-xr-x 1 jenkins jenkins 133M May 30 14:51 sync_diff_inspector -rwxr-xr-x 1 jenkins jenkins 199M May 30 14:51 tidb-server -rwxr-xr-x 1 jenkins jenkins 377M May 30 14:51 tiflash -rwxr-xr-x 1 jenkins jenkins 357M May 30 14:51 tikv-server [Pipeline] } Cache saved successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc) 3309751296 bytes in 62.49 secs (52963249 bytes/sec) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Tests) [Pipeline] parallel [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G00') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G01') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G02') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G03') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G04') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G05') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G06') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G07') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G08') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G09') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G10') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G11') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G12') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G13') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G14') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G15') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G16') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G17') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G18') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G19') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G20') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G21') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G00') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G01') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G02') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G03') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G04') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G05') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G06') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G07') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G08') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G09') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G10') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G11') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G12') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G13') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G14') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G15') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G16') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G17') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G18') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G19') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G20') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G21') [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-th7j8 Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-js3jc [Pipeline] node Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3 [Pipeline] node Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-mzktr Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-th7j8 is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-z8k57-5tzls --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "05209ad0ac540dcdb08316c83381d9f0c3d12a1e" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-z8k57" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-th7j8" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-th7j8" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-th7j8" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-th7j8 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test [Pipeline] { [Pipeline] checkout Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate The recommended git tool is: git [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6w171 Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-js3jc is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-6p9cv-4bt69 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "083e1b552fd337350ab614fb0c76bd465a13398f" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-6p9cv" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-js3jc" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-js3jc" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-js3jc" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-js3jc in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test [Pipeline] { [Pipeline] checkout Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git The recommended git tool is: git [Pipeline] podTemplate No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@28ff399f; decorates RemoteLauncher[hudson.remoting.Channel@bfe30ee:JNLP4-connect connection from 10.233.66.80/10.233.66.80:51970] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] { Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4168z [Pipeline] node Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@2eb5e1b3; decorates RemoteLauncher[hudson.remoting.Channel@58cc6dc5:JNLP4-connect connection from 10.233.127.192/10.233.127.192:35062] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t5729 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3 is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-37365-f4c49 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "4c2872cbd9d85073fcf3fba803b9f4a505cba865" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-37365" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test [Pipeline] { [Pipeline] checkout The recommended git tool is: git Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l00sh > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@540ef695; decorates RemoteLauncher[hudson.remoting.Channel@6cd502f8:JNLP4-connect connection from 10.233.105.26/10.233.105.26:47358] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" [Pipeline] withEnv [Pipeline] { Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] node [Pipeline] stage [Pipeline] { (Test) [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6w171 is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-kn6cm-7nd1z --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "9a1121fa487021d2e4d7e40a11928fa24e3b7218" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-kn6cm" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6w171" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6w171" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6w171" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6w171 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test [Pipeline] { [Pipeline] checkout [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-htrwd The recommended git tool is: git [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] cache > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-mzktr is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-np16g-lfjhh --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "caecf95b93d869ed26398029132232b9b2e2ea6b" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-np16g" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-mzktr" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-mzktr" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-mzktr" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-mzktr in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@7366b038; decorates RemoteLauncher[hudson.remoting.Channel@53a9084e:JNLP4-connect connection from 10.233.127.61/10.233.127.61:51692] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l00sh is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-w100l-h7vp0 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "adf8554a85a43856213e147836a66436235cd165" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-w100l" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l00sh" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l00sh" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l00sh" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l00sh in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t5729 is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-1d1v8-hrj24 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "04dd379d538120787aed3cba79052dce4e618871" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-1d1v8" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t5729" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t5729" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t5729" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t5729 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4168z is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-jw85k-2wjpv --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "ae577e00fd5f4fcd345df68582f71108f5060379" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-jw85k" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4168z" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4168z" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4168z" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4168z in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-htrwd is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-6sc1k-qkwgh --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "9544946783ce79017a315d35058f9ea0951d9c35" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-6sc1k" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-htrwd" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-htrwd" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-htrwd" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-htrwd in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc) 3309751296 bytes in 6.26 secs (528754128 bytes/sec) Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 [Pipeline] { [Pipeline] { [Pipeline] sh [Pipeline] { [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] { + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G00 Run cases: bdr_mode capture_suicide_while_balance_table syncpoint hang_sink_suicide server_config_compatibility changefeed_dup_error_restart kafka_big_messages kafka_compression kafka_messages kafka_sink_error_resume mq_sink_lost_callback mq_sink_dispatcher kafka_column_selector kafka_column_selector_avro lossy_ddl storage_csv_update PROW_JOB_ID=c548ef96-eb30-4461-bb19-187e41d862e3 JENKINS_NODE_COOKIE=c396b498-90e2-4aad-ac51-b02b0b5d9196 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/ GOLANG_VERSION=1.21.10 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-th7j8 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1796069089731416066","prowjobid":"c548ef96-eb30-4461-bb19-187e41d862e3","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#590 TEST_GROUP=G00 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089731416066 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-z8k57 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-z8k57 ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-th7j8 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-th7j8 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=590 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/bdr_mode/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] withEnv [Pipeline] { [Pipeline] checkout [Pipeline] { The recommended git tool is: git [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] checkout [Pipeline] container [Pipeline] { [Pipeline] { The recommended git tool is: git [Pipeline] checkout [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] podTemplate The recommended git tool is: git Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] { [Pipeline] node [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] node [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] cache No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@46722d97; decorates RemoteLauncher[hudson.remoting.Channel@37f7eedb:JNLP4-connect connection from 10.233.97.147/10.233.97.147:36314] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@74173077; decorates RemoteLauncher[hudson.remoting.Channel@1c86fefa:JNLP4-connect connection from 10.233.68.134/10.233.68.134:40806] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@24d7a4ee; decorates RemoteLauncher[hudson.remoting.Channel@1257d782:JNLP4-connect connection from 10.233.93.30/10.233.93.30:55310] will be ignored (a typical symptom is the Git executable not being run inside a designated container) No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@2db6fc21; decorates RemoteLauncher[hudson.remoting.Channel@7fae4fd5:JNLP4-connect connection from 10.233.105.206/10.233.105.206:41868] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Cloning the remote Git repository Using shallow clone with depth 1 Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@557cf042; decorates RemoteLauncher[hudson.remoting.Channel@7fcbeddb:JNLP4-connect connection from 10.233.93.99/10.233.93.99:38534] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4vpcx Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-gszd6 Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-9tzt0 Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l2n7g Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-nk3lw Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-lpxxn Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-p8f0f Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t2zx2 Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-hr7dq > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git start tidb cluster in /tmp/tidb_cdc_test/bdr_mode Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l2n7g is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-qwk3v-0jtc0 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "128e7bd5810d184fb2935b41b3d3e7b4445036d9" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-qwk3v" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l2n7g" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l2n7g" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l2n7g" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l2n7g in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-p8f0f is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-kjrs1-pn50f Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-6jbwn-6xrn0 Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-hr7dq is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-vx70x-003jc --- apiVersion: "v1" kind: "Pod" metadata: annotations: --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" labels: jenkins/jenkins-jenkins-agent: "true" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "1ba8515f963962b13cf8baa25f6c64c2fd8d00f4" jenkins/label-digest: "aa99a2db9f10961af84592a9ebaed549b6f86714" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-kjrs1" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-p8f0f" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-6jbwn" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w" namespace: "jenkins-tiflow" --- apiVersion: "v1" kind: "Pod" metadata: annotations: namespace: "jenkins-tiflow" buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "afb0283ac94ec1c679c0663a4f7377232c76184b" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-vx70x" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-hr7dq" namespace: "jenkins-tiflow" spec: affinity: values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-hr7dq" - name: "JENKINS_NAME" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-hr7dq" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" - mountPath: "/home/jenkins/agent" name: "workspace-volume" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" - emptyDir: medium: "" name: "workspace-volume" name: "workspace-volume" readOnly: false cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-p8f0f" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-p8f0f" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" requests: memory: "256Mi" cpu: "100m" volumeMounts: image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-hr7dq in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-p8f0f in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-lpxxn is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-wg70k-pg4s0 Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-gszd6 is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-df8hn-74ql8 Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4vpcx is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-bfvrw-rds5m --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" --- apiVersion: "v1" kind: "Pod" metadata: annotations: --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" labels: jenkins/jenkins-jenkins-agent: "true" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "2c348bcb254300b84161e04605a882bfe88025f8" jenkins/label-digest: "cd5379fd4849779652e7b8281e731cd736bcc614" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-df8hn" buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-gszd6" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-wg70k" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-lpxxn" namespace: "jenkins-tiflow" spec: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "ca3bd08123e5a9628d4c736055db8594679c09a0" - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" resources: limits: memory: "16Gi" cpu: "6" requests: jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-bfvrw" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4vpcx" namespace: "jenkins-tiflow" spec: affinity: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-gszd6" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-gszd6" - name: "JENKINS_AGENT_WORKDIR" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-lpxxn" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-lpxxn" - name: "JENKINS_AGENT_WORKDIR" - emptyDir: medium: "" name: "workspace-volume" memory: "128Mi" cpu: "100m" tty: true volumeMounts: value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" name: "report" resources: limits: memory: "256Mi" cpu: "100m" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - emptyDir: medium: "" name: "workspace-volume" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4vpcx" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4vpcx" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-gszd6 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-lpxxn in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4vpcx in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-nk3lw is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-xzf88-pj63h --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "ba2c608e6952539ebb191bc962330ca79ba7f66d" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-xzf88" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-nk3lw" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-nk3lw" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-nk3lw" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-9tzt0 is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-sdwtj-x4n6w --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "545842b4d79baa3afeec8f45c12d024cc0cbf3f6" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-sdwtj" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-9tzt0" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-9tzt0" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-9tzt0" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-9tzt0 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-nk3lw in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t2zx2 is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-55jnd-87h59 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "73cc1350ffb9cec5526dac1636c492d61e1add50" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-55jnd" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t2zx2" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t2zx2" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t2zx2" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t2zx2 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f211b45280010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-th7j8, pid:1194, start at 2024-05-30 14:53:55.421580796 +0800 CST m=+5.134802791 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:55:55.429 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:53:55.402 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:43:55.402 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f211b45280010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-th7j8, pid:1194, start at 2024-05-30 14:53:55.421580796 +0800 CST m=+5.134802791 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:55:55.429 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:53:55.402 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:43:55.402 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f211b45e40010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-th7j8, pid:1285, start at 2024-05-30 14:53:55.468555824 +0800 CST m=+5.096042449 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:55:55.476 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:53:55.449 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:43:55.449 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/bdr_mode/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/bdr_mode/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/bdr_mode/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/bdr_mode/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/bdr_mode/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Thu May 30 14:53:58 CST 2024] <<<<<< START cdc server in bdr_mode case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bdr_mode.25882590.out server --log-file /tmp/tidb_cdc_test/bdr_mode/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/bdr_mode/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:54:01 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/81ba0ff1-98e9-4aad-8e99-ece0e2d4edce {"id":"81ba0ff1-98e9-4aad-8e99-ece0e2d4edce","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc846a778b7 81ba0ff1-98e9-4aad-8e99-ece0e2d4edce /tidb/cdc/default/default/upstream/7374682302716834050 {"id":7374682302716834050,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/81ba0ff1-98e9-4aad-8e99-ece0e2d4edce {"id":"81ba0ff1-98e9-4aad-8e99-ece0e2d4edce","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc846a778b7 81ba0ff1-98e9-4aad-8e99-ece0e2d4edce /tidb/cdc/default/default/upstream/7374682302716834050 {"id":7374682302716834050,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/81ba0ff1-98e9-4aad-8e99-ece0e2d4edce {"id":"81ba0ff1-98e9-4aad-8e99-ece0e2d4edce","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc846a778b7 81ba0ff1-98e9-4aad-8e99-ece0e2d4edce /tidb/cdc/default/default/upstream/7374682302716834050 {"id":7374682302716834050,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Thu May 30 14:54:01 CST 2024] <<<<<< START cdc server in bdr_mode case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8400/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bdr_mode.26362638.out server --log-file /tmp/tidb_cdc_test/bdr_mode/cdcdown.log --log-level debug --data-dir /tmp/tidb_cdc_test/bdr_mode/cdc_datadown --cluster-id default --addr 127.0.0.1:8400 --pd http://127.0.0.1:2479 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8400/debug/info * About to connect() to 127.0.0.1 port 8400 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8400; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc) 3309751296 bytes in 20.42 secs (162062855 bytes/sec) [Pipeline] { [Pipeline] cache + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8400/debug/info * About to connect() to 127.0.0.1 port 8400 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8400 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8400 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:54:04 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d02519a1-8d5e-4725-9ba0-720449f42f4c {"id":"d02519a1-8d5e-4725-9ba0-720449f42f4c","address":"127.0.0.1:8400","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/35118fc846a7a672 d02519a1-8d5e-4725-9ba0-720449f42f4c /tidb/cdc/default/default/upstream/7374682303078587254 {"id":7374682303078587254,"pd-endpoints":"http://127.0.0.1:2479,http://127.0.0.1:2479","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d02519a1-8d5e-4725-9ba0-720449f42f4c {"id":"d02519a1-8d5e-4725-9ba0-720449f42f4c","address":"127.0.0.1:8400","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/35118fc846a7a672 d02519a1-8d5e-4725-9ba0-720449f42f4c /tidb/cdc/default/default/upstream/7374682303078587254 {"id":7374682303078587254,"pd-endpoints":"http://127.0.0.1:2479,http://127.0.0.1:2479","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d02519a1-8d5e-4725-9ba0-720449f42f4c {"id":"d02519a1-8d5e-4725-9ba0-720449f42f4c","address":"127.0.0.1:8400","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/35118fc846a7a672 d02519a1-8d5e-4725-9ba0-720449f42f4c /tidb/cdc/default/default/upstream/7374682303078587254 {"id":7374682303078587254,"pd-endpoints":"http://127.0.0.1:2479,http://127.0.0.1:2479","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table bdr_mode.t1 exists table bdr_mode.t1 exists + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bdr_mode.cli.2709.out cli changefeed create --sink-uri=mysql://root@127.0.0.1:3306 -c test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/bdr_mode/conf/up.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7374682302716834050,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306","create_time":"2024-05-30T14:54:05.787605292+08:00","start_ts":450114891468767233,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":true,"bdr_mode":true,"sync_point_interval":30000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114891468767233,"checkpoint_ts":450114891468767233,"checkpoint_time":"2024-05-30 14:54:05.703"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bdr_mode.cli.2748.out cli changefeed create --sink-uri=mysql://root@127.0.0.1:4000 -c test-2 --server http://127.0.0.1:8400 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/bdr_mode/conf/down.toml Create changefeed successfully! ID: test-2 Info: {"upstream_id":7374682303078587254,"namespace":"default","id":"test-2","sink_uri":"mysql://root@127.0.0.1:4000","create_time":"2024-05-30T14:54:07.649626594+08:00","start_ts":450114891939577861,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":true,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114891939577861,"checkpoint_ts":450114891939577861,"checkpoint_time":"2024-05-30 14:54:07.499"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x table tidb_cdc.syncpoint_v1 exists table tidb_cdc.syncpoint_v1 does not exists table bdr_mode.finish_mark exists table bdr_mode.finish_mark exists check diff failed 1-th time, retry later check diff failed 2-th time, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc) 3309751296 bytes in 5.60 secs (591277378 bytes/sec) [Pipeline] { [Pipeline] cache check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 14:54:16 CST 2024] <<<<<< run test case bdr_mode success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/capture_suicide_while_balance_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/capture_suicide_while_balance_table Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc) 3309751296 bytes in 15.77 secs (209854024 bytes/sec) [Pipeline] { [Pipeline] sh [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G01 Run cases: http_api http_api_tls api_v2 kafka_simple_basic kafka_simple_basic_avro kafka_simple_handle_key_only kafka_simple_handle_key_only_avro kafka_simple_claim_check kafka_simple_claim_check_avro canal_json_adapter_compatibility canal_json_basic multi_topics avro_basic canal_json_handle_key_only open_protocol_handle_key_only canal_json_claim_check open_protocol_claim_check canal_json_storage_basic canal_json_storage_partition_table multi_tables_ddl PROW_JOB_ID=c548ef96-eb30-4461-bb19-187e41d862e3 JENKINS_NODE_COOKIE=294979fc-619b-40bc-ac03-d8ddc8c738fc BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/ GOLANG_VERSION=1.21.10 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-js3jc HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1796069089731416066","prowjobid":"c548ef96-eb30-4461-bb19-187e41d862e3","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#590 TEST_GROUP=G01 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089731416066 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-6p9cv GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-js3jc ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-6p9cv GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-js3jc GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=590 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/http_api/run.sh using Sink-Type: mysql... <<================= [Pipeline] sh WARNING: Running pip install with root privileges is generally not a good idea. Try `__main__.py install --user` instead. Collecting requests==2.26.0 + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G04 Run cases: foreign_key ddl_puller_lag ddl_only_block_related_table changefeed_auto_stop PROW_JOB_ID=c548ef96-eb30-4461-bb19-187e41d862e3 JENKINS_NODE_COOKIE=56719ebc-d369-4610-a4eb-08eb98d81662 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/ GOLANG_VERSION=1.21.10 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6w171 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1796069089731416066","prowjobid":"c548ef96-eb30-4461-bb19-187e41d862e3","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#590 TEST_GROUP=G04 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089731416066 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-kn6cm GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6w171 ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-kn6cm GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6w171 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=590 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/foreign_key/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Downloading https://files.pythonhosted.org/packages/92/96/144f70b972a9c0eabbd4391ef93ccd49d0f2747f4f6a2a2738e99e5adc65/requests-2.26.0-py2.py3-none-any.whl (62kB) Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Requirement already up-to-date: charset-normalizer~=2.0.0; python_version >= "3" in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) Collecting certifi>=2017.4.17 (from requests==2.26.0) Downloading https://files.pythonhosted.org/packages/ba/06/a07f096c664aeb9f01624f858c3add0a4e913d6c96257acb4fce61e7de14/certifi-2024.2.2-py3-none-any.whl (163kB) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Collecting urllib3<1.27,>=1.21.1 (from requests==2.26.0) Downloading https://files.pythonhosted.org/packages/b0/53/aa91e163dcfd1e5b82d8a890ecf13314e3e149c05270cc644581f77f17fd/urllib3-1.26.18-py2.py3-none-any.whl (143kB) [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate Collecting idna<4,>=2.5; python_version >= "3" (from requests==2.26.0) Downloading https://files.pythonhosted.org/packages/e5/3e/741d8c82801c347547f8a2a06aa57dbb1992be9e948df2ea0eda2c8b79e8/idna-3.7-py3-none-any.whl (66kB) Installing collected packages: certifi, urllib3, idna, requests Found existing installation: certifi 2023.5.7 Uninstalling certifi-2023.5.7: Successfully uninstalled certifi-2023.5.7 Found existing installation: urllib3 1.26.16 Uninstalling urllib3-1.26.16: Successfully uninstalled urllib3-1.26.16 Found existing installation: idna 3.4 Uninstalling idna-3.4: Successfully uninstalled idna-3.4 Found existing installation: requests 2.27.1 Uninstalling requests-2.27.1: Successfully uninstalled requests-2.27.1 Successfully installed certifi-2024.2.2 idna-3.7 requests-2.26.0 urllib3-1.26.18 start tidb cluster in /tmp/tidb_cdc_test/foreign_key Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... [Pipeline] { The 1 times to try to start tidb cluster... [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] { + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G02 Run cases: consistent_replicate_ddl consistent_replicate_gbk consistent_replicate_nfs consistent_replicate_storage_file consistent_replicate_storage_file_large_value consistent_replicate_storage_s3 consistent_partition_table kafka_big_txn_v2 kafka_big_messages_v2 multi_tables_ddl_v2 multi_topics_v2 storage_cleanup csv_storage_basic csv_storage_multi_tables_ddl csv_storage_partition_table PROW_JOB_ID=c548ef96-eb30-4461-bb19-187e41d862e3 JENKINS_NODE_COOKIE=0e76e633-7c31-420e-ba37-919de4e6451c BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/ GOLANG_VERSION=1.21.10 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1796069089731416066","prowjobid":"c548ef96-eb30-4461-bb19-187e41d862e3","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#590 TEST_GROUP=G02 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089731416066 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-37365 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-37365 ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=590 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/consistent_replicate_ddl/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] withEnv [Pipeline] { Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-cvg08 Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-bcfm9 Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-0vnff [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f211de6380006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-th7j8, pid:3905, start at 2024-05-30 14:54:38.484363765 +0800 CST m=+5.151477154 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:56:38.491 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:54:38.478 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:44:38.478 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f211de6380006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-th7j8, pid:3905, start at 2024-05-30 14:54:38.484363765 +0800 CST m=+5.151477154 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:56:38.491 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:54:38.478 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:44:38.478 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f211de9340014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-th7j8, pid:3952, start at 2024-05-30 14:54:38.692696644 +0800 CST m=+5.226766127 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:56:38.700 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:54:38.669 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:44:38.669 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/capture_suicide_while_balance_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/capture_suicide_while_balance_table/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/capture_suicide_while_balance_table/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/capture_suicide_while_balance_table/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/capture_suicide_while_balance_table/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Pipeline] checkout start tidb cluster in /tmp/tidb_cdc_test/http_api Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git The recommended git tool is: git [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [Pipeline] stage [Pipeline] { (Test) No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@26096120; decorates RemoteLauncher[hudson.remoting.Channel@7041b980:JNLP4-connect connection from 10.233.106.86/10.233.106.86:33246] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@7098d9cc; decorates RemoteLauncher[hudson.remoting.Channel@526a7624:JNLP4-connect connection from 10.233.93.68/10.233.93.68:46116] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 start tidb cluster in /tmp/tidb_cdc_test/consistent_replicate_ddl Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@211a3853; decorates RemoteLauncher[hudson.remoting.Channel@9b003c9:JNLP4-connect connection from 10.233.93.246/10.233.93.246:59346] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@2359d3c6; decorates RemoteLauncher[hudson.remoting.Channel@6eb4aa74:JNLP4-connect connection from 10.233.97.215/10.233.97.215:49902] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@69752354; decorates RemoteLauncher[hudson.remoting.Channel@6e281859:JNLP4-connect connection from 10.233.97.4/10.233.97.4:41482] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@1b0acd25; decorates RemoteLauncher[hudson.remoting.Channel@3576f97:JNLP4-connect connection from 10.233.68.175/10.233.68.175:41104] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@2a784bbb; decorates RemoteLauncher[hudson.remoting.Channel@71bd2301:JNLP4-connect connection from 10.233.100.214/10.233.100.214:55410] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@4923808e; decorates RemoteLauncher[hudson.remoting.Channel@a922549:JNLP4-connect connection from 10.233.107.120/10.233.107.120:42448] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@384d8f5d; decorates RemoteLauncher[hudson.remoting.Channel@75451d48:JNLP4-connect connection from 10.233.107.99/10.233.107.99:60708] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@1333ffde; decorates RemoteLauncher[hudson.remoting.Channel@2fe592f0:JNLP4-connect connection from 10.233.107.157/10.233.107.157:43528] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] stage [Pipeline] { (Test) Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] stage Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 [Pipeline] { (Test) Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Thu May 30 14:54:41 CST 2024] <<<<<< START cdc server in capture_suicide_while_balance_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_suicide_while_balance_table.51935195.out server --log-file /tmp/tidb_cdc_test/capture_suicide_while_balance_table/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/capture_suicide_while_balance_table/cdc_data1 --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-bcfm9 is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-gqzgb-q0s6t Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-0vnff is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-961q0-f50jp --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "039a8fde8bca2b7e6967b98ab269f35cae459977" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-gqzgb" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-bcfm9" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" --- apiVersion: "v1" kind: "Pod" metadata: annotations: operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "5b1aebee276991e4e608f8bca9b41a0834f64dfa" - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-961q0" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-0vnff" namespace: "jenkins-tiflow" limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-bcfm9" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-bcfm9" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-0vnff" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-0vnff" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" [Pipeline] timeout Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-bcfm9 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Timeout set to expire in 40 min Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-0vnff in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-cvg08 is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-v14kh-3h9r2 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "7852c6d3d97b353af574c5a5a5b7c907fb7bc833" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-v14kh" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-cvg08" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-cvg08" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-cvg08" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-cvg08 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] { Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Avoid second fetch [Pipeline] { Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [Pipeline] { Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" [Pipeline] { ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] { [Pipeline] { Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" [Pipeline] { > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 [Pipeline] { > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 [Pipeline] withEnv [Pipeline] { [Pipeline] checkout [Pipeline] checkout The recommended git tool is: git + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:54:44 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/43c0fc77-73ee-4bd4-bf37-2935e7402f44 {"id":"43c0fc77-73ee-4bd4-bf37-2935e7402f44","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc847517fbc 43c0fc77-73ee-4bd4-bf37-2935e7402f44 /tidb/cdc/default/default/upstream/7374682489134078232 {"id":7374682489134078232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/43c0fc77-73ee-4bd4-bf37-2935e7402f44 {"id":"43c0fc77-73ee-4bd4-bf37-2935e7402f44","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc847517fbc 43c0fc77-73ee-4bd4-bf37-2935e7402f44 /tidb/cdc/default/default/upstream/7374682489134078232 {"id":7374682489134078232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/43c0fc77-73ee-4bd4-bf37-2935e7402f44 {"id":"43c0fc77-73ee-4bd4-bf37-2935e7402f44","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc847517fbc 43c0fc77-73ee-4bd4-bf37-2935e7402f44 /tidb/cdc/default/default/upstream/7374682489134078232 {"id":7374682489134078232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Thu May 30 14:54:44 CST 2024] <<<<<< START cdc server in capture_suicide_while_balance_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=1*return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_suicide_while_balance_table.52465248.out server --log-file /tmp/tidb_cdc_test/capture_suicide_while_balance_table/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/capture_suicide_while_balance_table/cdc_data2 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] checkout The recommended git tool is: git The recommended git tool is: git [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] cache Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@78d53f28; decorates RemoteLauncher[hudson.remoting.Channel@44c94c20:JNLP4-connect connection from 10.233.107.152/10.233.107.152:52576] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@46aa4d7; decorates RemoteLauncher[hudson.remoting.Channel@3e5574ee:JNLP4-connect connection from 10.233.126.94/10.233.126.94:50214] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@7046225f; decorates RemoteLauncher[hudson.remoting.Channel@25010db9:JNLP4-connect connection from 10.233.127.23/10.233.127.23:57516] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:54:47 GMT < Content-Length: 867 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/43c0fc77-73ee-4bd4-bf37-2935e7402f44 {"id":"43c0fc77-73ee-4bd4-bf37-2935e7402f44","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/7535143a-ce0a-43f8-ba78-93f892f660cf {"id":"7535143a-ce0a-43f8-ba78-93f892f660cf","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc847517fbc 43c0fc77-73ee-4bd4-bf37-2935e7402f44 /tidb/cdc/default/__cdc_meta__/owner/22318fc847517fe6 7535143a-ce0a-43f8-ba78-93f892f660cf /tidb/cdc/default/default/upstream/7374682489134078232 {"id":7374682489134078232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/43c0fc77-73ee-4bd4-bf37-2935e7402f44 {"id":"43c0fc77-73ee-4bd4-bf37-2935e7402f44","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/7535143a-ce0a-43f8-ba78-93f892f660cf {"id":"7535143a-ce0a-43f8-ba78-93f892f660cf","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc847517fbc 43c0fc77-73ee-4bd4-bf37-2935e7402f44 /tidb/cdc/default/__cdc_meta__/owner/22318fc847517fe6 7535143a-ce0a-43f8-ba78-93f892f660cf /tidb/cdc/default/default/upstream/7374682489134078232 {"id":7374682489134078232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/43c0fc77-73ee-4bd4-bf37-2935e7402f44 {"id":"43c0fc77-73ee-4bd4-bf37-2935e7402f44","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/7535143a-ce0a-43f8-ba78-93f892f660cf {"id":"7535143a-ce0a-43f8-ba78-93f892f660cf","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc847517fbc 43c0fc77-73ee-4bd4-bf37-2935e7402f44 /tidb/cdc/default/__cdc_meta__/owner/22318fc847517fe6 7535143a-ce0a-43f8-ba78-93f892f660cf /tidb/cdc/default/default/upstream/7374682489134078232 {"id":7374682489134078232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f211e6e640017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6w171, pid:1192, start at 2024-05-30 14:54:47.220488721 +0800 CST m=+5.187656234 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:56:47.228 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:54:47.193 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:44:47.193 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f211e6e640017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6w171, pid:1192, start at 2024-05-30 14:54:47.220488721 +0800 CST m=+5.187656234 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:56:47.228 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:54:47.193 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:44:47.193 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f211e70940002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6w171, pid:1266, start at 2024-05-30 14:54:47.334673373 +0800 CST m=+5.252987211 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:56:47.342 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:54:47.333 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:44:47.333 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/foreign_key/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/foreign_key/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.cli.2563.out cli tso query --pd=http://127.0.0.1:2379 Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 table capture_suicide_while_balance_table.t1 exists table capture_suicide_while_balance_table.t2 not exists for 1-th check, retry later + set +x + tso='450114903249780737 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114903249780737 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 14:54:52 CST 2024] <<<<<< START cdc server in foreign_key case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.25972599.out server --log-file /tmp/tidb_cdc_test/foreign_key/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/foreign_key/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f211eb00c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-js3jc, pid:1281, start at 2024-05-30 14:54:51.420383439 +0800 CST m=+5.389473594 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:56:51.428 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:54:51.395 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:44:51.395 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table capture_suicide_while_balance_table.t2 exists table capture_suicide_while_balance_table.t3 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f211eb00c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-js3jc, pid:1281, start at 2024-05-30 14:54:51.420383439 +0800 CST m=+5.389473594 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:56:51.428 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:54:51.395 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:44:51.395 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f211eaff40014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-js3jc, pid:1340, start at 2024-05-30 14:54:51.410083592 +0800 CST m=+5.320171890 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:56:51.416 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:54:51.389 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:44:51.389 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/http_api/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/http_api/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/http_api/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/http_api/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/http_api/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f211edb180013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3, pid:1189, start at 2024-05-30 14:54:54.179170756 +0800 CST m=+7.885204271 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:56:54.186 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:54:54.150 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:44:54.150 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f211edb180013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3, pid:1189, start at 2024-05-30 14:54:54.179170756 +0800 CST m=+7.885204271 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:56:54.186 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:54:54.150 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:44:54.150 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f211eccf80014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3, pid:1264, start at 2024-05-30 14:54:53.26680201 +0800 CST m=+6.907243283 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:56:53.274 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:54:53.246 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:44:53.246 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_replicate_ddl/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_replicate_ddl/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_replicate_ddl/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/consistent_replicate_ddl/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/consistent_replicate_ddl/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table capture_suicide_while_balance_table.t3 exists table capture_suicide_while_balance_table.t4 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:54:55 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/751ff4bb-5a9c-4779-ae3a-e7148a612d08 {"id":"751ff4bb-5a9c-4779-ae3a-e7148a612d08","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8477194bf 751ff4bb-5a9c-4779-ae3a-e7148a612d08 /tidb/cdc/default/default/upstream/7374682523896563788 {"id":7374682523896563788,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/751ff4bb-5a9c-4779-ae3a-e7148a612d08 {"id":"751ff4bb-5a9c-4779-ae3a-e7148a612d08","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8477194bf 751ff4bb-5a9c-4779-ae3a-e7148a612d08 /tidb/cdc/default/default/upstream/7374682523896563788 {"id":7374682523896563788,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/751ff4bb-5a9c-4779-ae3a-e7148a612d08 {"id":"751ff4bb-5a9c-4779-ae3a-e7148a612d08","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8477194bf 751ff4bb-5a9c-4779-ae3a-e7148a612d08 /tidb/cdc/default/default/upstream/7374682523896563788 {"id":7374682523896563788,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.cli.2641.out cli changefeed create --start-ts=450114903249780737 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: 5a114816-525b-48d3-ac40-b47c5ec8baad Info: {"upstream_id":7374682523896563788,"namespace":"default","id":"5a114816-525b-48d3-ac40-b47c5ec8baad","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T14:54:55.618724804+08:00","start_ts":450114903249780737,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114903249780737,"checkpoint_ts":450114903249780737,"checkpoint_time":"2024-05-30 14:54:50.644"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [Thu May 30 14:54:56 CST 2024] <<<<<< START cdc server in http_api case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_api.26262628.out server --log-file /tmp/tidb_cdc_test/http_api/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_api/cdc_data1 --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Thu May 30 14:54:56 CST 2024] <<<<<< START cdc server in consistent_replicate_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_ddl.25662568.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_ddl/cdcconsistent_replicate_ddl.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_ddl/cdc_dataconsistent_replicate_ddl.server1 --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x table capture_suicide_while_balance_table.t4 exists + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:54:59 GMT < Content-Length: 657 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/aa3fb5c3-921a-40de-969b-7564171ff2cd {"id":"aa3fb5c3-921a-40de-969b-7564171ff2cd","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84786df35 aa3fb5c3-921a-40de-969b-7564171ff2cd /tidb/cdc/default/default/upstream/7374682541054098141 {"id":7374682541054098141,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/aa3fb5c3-921a-40de-969b-7564171ff2cd {"id":"aa3fb5c3-921a-40de-969b-7564171ff2cd","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84786df35 aa3fb5c3-921a-40de-969b-7564171ff2cd /tidb/cdc/default/default/upstream/7374682541054098141 {"id":7374682541054098141,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/aa3fb5c3-921a-40de-969b-7564171ff2cd {"id":"aa3fb5c3-921a-40de-969b-7564171ff2cd","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84786df35 aa3fb5c3-921a-40de-969b-7564171ff2cd /tidb/cdc/default/default/upstream/7374682541054098141 {"id":7374682541054098141,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 140 100 35 100 105 9041 27124 --:--:-- --:--:-- --:--:-- 35000 { "status": true, "message": "" + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:54:59 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e8fac868-7100-4215-8c76-c6f0c9cdc6a6 {"id":"e8fac868-7100-4215-8c76-c6f0c9cdc6a6","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8478427bc e8fac868-7100-4215-8c76-c6f0c9cdc6a6 /tidb/cdc/default/default/upstream/7374682542777058856 {"id":7374682542777058856,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e8fac868-7100-4215-8c76-c6f0c9cdc6a6 {"id":"e8fac868-7100-4215-8c76-c6f0c9cdc6a6","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8478427bc e8fac868-7100-4215-8c76-c6f0c9cdc6a6 /tidb/cdc/default/default/upstream/7374682542777058856 {"id":7374682542777058856,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e8fac868-7100-4215-8c76-c6f0c9cdc6a6 {"id":"e8fac868-7100-4215-8c76-c6f0c9cdc6a6","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8478427bc e8fac868-7100-4215-8c76-c6f0c9cdc6a6 /tidb/cdc/default/default/upstream/7374682542777058856 {"id":7374682542777058856,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ***************** properties ***************** "dotransactions"="false" "threadcount"="10" "mysql.db"="consistent_replicate_ddl" "insertproportion"="0" "requestdistribution"="uniform" "mysql.host"="127.0.0.1" "readproportion"="0" "updateproportion"="0" "scanproportion"="0" "operationcount"="0" "workload"="core" "fieldcount"="3" "mysql.port"="4000" "mysql.user"="root" "readallfields"="true" "recordcount"="50" ********************************************** Run finished, takes 11.332481ms INSERT - Takes(s): 0.0, Count: 50, OPS: 8774.9, Avg(us): 2082, Min(us): 964, Max(us): 5900, 95th(us): 6000, 99th(us): 6000 table consistent_replicate_ddl.usertable1 not exists for 1-th check, retry later "is-owner": true, run task successfully }lease 22318fc847517fe6 revoked check diff successfully owner pid: 2631 owner id aa3fb5c3-921a-40de-969b-7564171ff2cd default [Thu May 30 14:55:02 CST 2024] <<<<<< START cdc server in http_api case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_api.27482750.out server --log-file /tmp/tidb_cdc_test/http_api/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_api/cdc_data2 --cluster-id default --addr 127.0.0.1:8301 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 1-th time... table consistent_replicate_ddl.usertable1 not exists for 2-th check, retry later wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 14:55:03 CST 2024] <<<<<< run test case capture_suicide_while_balance_table success! >>>>>> table consistent_replicate_ddl.usertable1 not exists for 3-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:55:05 GMT < Content-Length: 911 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2762a262-6ae4-45f3-99ca-a7909c3e59d5 {"id":"2762a262-6ae4-45f3-99ca-a7909c3e59d5","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/aa3fb5c3-921a-40de-969b-7564171ff2cd {"id":"aa3fb5c3-921a-40de-969b-7564171ff2cd","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84786df35 aa3fb5c3-921a-40de-969b-7564171ff2cd /tidb/cdc/default/__cdc_meta__/owner/22318fc84786df60 2762a262-6ae4-45f3-99ca-a7909c3e59d5 /tidb/cdc/default/default/upstream/7374682541054098141 {"id":7374682541054098141,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2762a262-6ae4-45f3-99ca-a7909c3e59d5 {"id":"2762a262-6ae4-45f3-99ca-a7909c3e59d5","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/aa3fb5c3-921a-40de-969b-7564171ff2cd {"id":"aa3fb5c3-921a-40de-969b-7564171ff2cd","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84786df35 aa3fb5c3-921a-40de-969b-7564171ff2cd /tidb/cdc/default/__cdc_meta__/owner/22318fc84786df60 2762a262-6ae4-45f3-99ca-a7909c3e59d5 /tidb/cdc/default/default/upstream/7374682541054098141 {"id":7374682541054098141,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2762a262-6ae4-45f3-99ca-a7909c3e59d5 {"id":"2762a262-6ae4-45f3-99ca-a7909c3e59d5","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/aa3fb5c3-921a-40de-969b-7564171ff2cd {"id":"aa3fb5c3-921a-40de-969b-7564171ff2cd","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84786df35 aa3fb5c3-921a-40de-969b-7564171ff2cd /tidb/cdc/default/__cdc_meta__/owner/22318fc84786df60 2762a262-6ae4-45f3-99ca-a7909c3e59d5 /tidb/cdc/default/default/upstream/7374682541054098141 {"id":7374682541054098141,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep -v "aa3fb5c3-921a-40de-969b-7564171ff2cd default" | grep id Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc) 3309751296 bytes in 17.63 secs (187754595 bytes/sec) [Pipeline] { [Pipeline] cache table foreign_key.finish_mark not exists for 1-th check, retry later "id": "2762a262-6ae4-45f3-99ca-a7909c3e59d5", run task successfully table consistent_replicate_ddl.usertable1 not exists for 4-th check, retry later table foreign_key.finish_mark not exists for 2-th check, retry later capture_id: 2762a262-6ae4-45f3-99ca-a7909c3e59d5 pass test: check health pass test: get status pass test: create changefeed table consistent_replicate_ddl.usertable1 exists table consistent_replicate_ddl.usertable2 not exists for 1-th check, retry later table foreign_key.finish_mark not exists for 3-th check, retry later table consistent_replicate_ddl.usertable2 not exists for 2-th check, retry later table test.simple1 not exists for 1-th check, retry later table foreign_key.finish_mark not exists for 4-th check, retry later table consistent_replicate_ddl.usertable2 exists table consistent_replicate_ddl.usertable3 not exists for 1-th check, retry later table test.simple1 not exists for 2-th check, retry later table foreign_key.finish_mark not exists for 5-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/syncpoint/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc) 3309751296 bytes in 5.49 secs (603051110 bytes/sec) [Pipeline] { [Pipeline] cache table consistent_replicate_ddl.usertable3 not exists for 2-th check, retry later table foreign_key.finish_mark not exists for 6-th check, retry later table test.simple1 exists pass test: list changefeed start tidb cluster in /tmp/tidb_cdc_test/syncpoint Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... pass test: get changefeed pass test: pause changefeed table consistent_replicate_ddl.usertable3 exists table consistent_replicate_ddl.usertable_bak not exists for 1-th check, retry later table foreign_key.finish_mark not exists for 7-th check, retry later pass test: update changefeed wait process 2631 exit for 1-th time... wait process 2631 exit for 2-th time... wait process 2631 exit for 3-th time... table consistent_replicate_ddl.usertable_bak not exists for 2-th check, retry later /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (2631) - No such process wait process 2631 exit for 4-th time... process 2631 already exit cdc.test cli capture list --server http://127.0.0.1:8301 2>&1 |grep 2762a262-6ae4-45f3-99ca-a7909c3e59d5 -A1 | grep '"is-owner": true' Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table foreign_key.finish_mark not exists for 8-th check, retry later "is-owner": true, run task successfully [Thu May 30 14:55:20 CST 2024] <<<<<< START cdc server in http_api case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_api.29682970.out server --log-file /tmp/tidb_cdc_test/http_api/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_api/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table consistent_replicate_ddl.usertable_bak exists Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table foreign_key.finish_mark not exists for 9-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc) 3309751296 bytes in 5.96 secs (555744190 bytes/sec) [Pipeline] { [Pipeline] cache ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table foreign_key.finish_mark exists check diff successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:55:23 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/changefeed-test1 {UpstreamID:7374682541054098141 Namespace:default ID:changefeed-test1 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:55:08.588312394 +0800 CST StartTs:450114907929313290 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037499e0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114907942158350} {CheckpointTs:450114908125659165 MinTableBarrierTs:450114911756353544 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114911769460741, checkpointTs: 450114911756353544, state: Replicating span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 450114911769460741, checkpointTs: 450114911756353544, state: Replicating changefeedID: default/changefeed-test3 {UpstreamID:7374682541054098141 Namespace:default ID:changefeed-test3 SinkURI:blackhole:// CreateTime:2024-05-30 14:55:08.804912541 +0800 CST StartTs:450114907994587147 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003654bd0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114908007694339} {CheckpointTs:450114908125659165 MinTableBarrierTs:450114911756353544 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114908125659165, checkpointTs: 450114908125659165, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2762a262-6ae4-45f3-99ca-a7909c3e59d5 {"id":"2762a262-6ae4-45f3-99ca-a7909c3e59d5","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/2be5bc53-8a4d-4299-bc73-1a1df84b2d42 {"id":"2be5bc53-8a4d-4299-bc73-1a1df84b2d42","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84786df60 2762a262-6ae4-45f3-99ca-a7909c3e59d5 /tidb/cdc/default/__cdc_meta__/owner/22318fc84786dfdb 2be5bc53-8a4d-4299-bc73-1a1df84b2d42 /tidb/cdc/default/default/changefeed/info/changefeed-test1 {"upstream-id":7374682541054098141,"namespace":"default","changefeed-id":"changefeed-test1","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:55:08.588312394+08:00","start-ts":450114907929313290,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114907942158350} /tidb/cdc/default/default/changefeed/info/changefeed-test2 {"upstream-id":7374682541054098141,"namespace":"default","changefeed-id":"changefeed-test2","sink-uri":"blackhole://","create-time":"2024-05-30T14:55:08.699975479+08:00","start-ts":450114907968372745,"target-ts":0,"admin-job-type":1,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":32},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"stopped","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114910065786886} /tidb/cdc/default/default/changefeed/info/changefeed-test3 {"upstream-id":7374682541054098141,"namespace":"default","changefeed-id":"changefeed-test3","sink-uri":"blackhole://","create-time":"2024-05-30T14:55:08.804912541+08:00","start-ts":450114907994587147,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114908007694339} /tidb/cdc/default/default/changefeed/status/changefeed-test1 {"checkpoint-ts":450114908125659165,"min-table-barrier-ts":450114911756353544,"admin-job-type":0} /tidb/cdc/default/default/changefeed/status/changefeed-test2 {"checkpoint-ts":450114908125659165,"min-table-barrier-ts":450114909921345542,"admin-job-type":1} /tidb/cdc/default/default/changefeed/status/changefeed-test3 {"checkpoint-ts":450114908125659165,"min-table-barrier-ts":450114911756353544,"admin-job-type":0} /tidb/cdc/default/default/task/position/2762a262-6ae4-45f3-99ca-a7909c3e59d5/changefeed-test1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/2762a262-6ae4-45f3-99ca-a7909c3e59d5/changefeed-test3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/2be5bc53-8a4d-4299-bc73-1a1df84b2d42/changefeed-test1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/2be5bc53-8a4d-4299-bc73-1a1df84b2d42/changefeed-test3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374682541054098141 {"id":7374682541054098141,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/changefeed-test1 {UpstreamID:7374682541054098141 Namespace:default ID:changefeed-test1 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:55:08.588312394 +0800 CST StartTs:450114907929313290 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037499e0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114907942158350} {CheckpointTs:450114908125659165 MinTableBarrierTs:450114911756353544 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114911769460741, checkpointTs: 450114911756353544, state: Replicating span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 450114911769460741, checkpointTs: 450114911756353544, state: Replicating changefeedID: default/changefeed-test3 {UpstreamID:7374682541054098141 Namespace:default ID:changefeed-test3 SinkURI:blackhole:// CreateTime:2024-05-30 14:55:08.804912541 +0800 CST StartTs:450114907994587147 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003654bd0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114908007694339} {CheckpointTs:450114908125659165 MinTableBarrierTs:450114911756353544 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114908125659165, checkpointTs: 450114908125659165, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2762a262-6ae4-45f3-99ca-a7909c3e59d5 {"id":"2762a262-6ae4-45f3-99ca-a7909c3e59d5","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/2be5bc53-8a4d-4299-bc73-1a1df84b2d42 {"id":"2be5bc53-8a4d-4299-bc73-1a1df84b2d42","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84786df60 2762a262-6ae4-45f3-99ca-a7909c3e59d5 /tidb/cdc/default/__cdc_meta__/owner/22318fc84786dfdb 2be5bc53-8a4d-4299-bc73-1a1df84b2d42 /tidb/cdc/default/default/changefeed/info/changefeed-test1 {"upstream-id":7374682541054098141,"namespace":"default","changefeed-id":"changefeed-test1","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:55:08.588312394+08:00","start-ts":450114907929313290,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114907942158350} /tidb/cdc/default/default/changefeed/info/changefeed-test2 {"upstream-id":7374682541054098141,"namespace":"default","changefeed-id":"changefeed-test2","sink-uri":"blackhole://","create-time":"2024-05-30T14:55:08.699975479+08:00","start-ts":450114907968372745,"target-ts":0,"admin-job-type":1,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":32},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"stopped","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114910065786886} /tidb/cdc/default/default/changefeed/info/changefeed-test3 {"upstream-id":7374682541054098141,"namespace":"default","changefeed-id":"changefeed-test3","sink-uri":"blackhole://","create-time":"2024-05-30T14:55:08.804912541+08:00","start-ts":450114907994587147,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114908007694339} /tidb/cdc/default/default/changefeed/status/changefeed-test1 {"checkpoint-ts":450114908125659165,"min-table-barrier-ts":450114911756353544,"admin-job-type":0} /tidb/cdc/default/default/changefeed/status/changefeed-test2 {"checkpoint-ts":450114908125659165,"min-table-barrier-ts":450114909921345542,"admin-job-type":1} /tidb/cdc/default/default/changefeed/status/changefeed-test3 {"checkpoint-ts":450114908125659165,"min-table-barrier-ts":450114911756353544,"admin-job-type":0} /tidb/cdc/default/default/task/position/2762a262-6ae4-45f3-99ca-a7909c3e59d5/changefeed-test1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/2762a262-6ae4-45f3-99ca-a7909c3e59d5/changefeed-test3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/2be5bc53-8a4d-4299-bc73-1a1df84b2d42/changefeed-test1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/2be5bc53-8a4d-4299-bc73-1a1df84b2d42/changefeed-test3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374682541054098141 {"id":7374682541054098141,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/changefeed-test1 {UpstreamID:7374682541054098141 Namespace:default ID:changefeed-test1 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:55:08.588312394 +0800 CST StartTs:450114907929313290 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037499e0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114907942158350} {CheckpointTs:450114908125659165 MinTableBarrierTs:450114911756353544 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114911769460741, checkpointTs: 450114911756353544, state: Replicating span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 450114911769460741, checkpointTs: 450114911756353544, state: Replicating changefeedID: default/changefeed-test3 {UpstreamID:7374682541054098141 Namespace:default ID:changefeed-test3 SinkURI:blackhole:// CreateTime:2024-05-30 14:55:08.804912541 +0800 CST StartTs:450114907994587147 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003654bd0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114908007694339} {CheckpointTs:450114908125659165 MinTableBarrierTs:450114911756353544 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114908125659165, checkpointTs: 450114908125659165, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2762a262-6ae4-45f3-99ca-a7909c3e59d5 {"id":"2762a262-6ae4-45f3-99ca-a7909c3e59d5","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/2be5bc53-8a4d-4299-bc73-1a1df84b2d42 {"id":"2be5bc53-8a4d-4299-bc73-1a1df84b2d42","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84786df60 2762a262-6ae4-45f3-99ca-a7909c3e59d5 /tidb/cdc/default/__cdc_meta__/owner/22318fc84786dfdb 2be5bc53-8a4d-4299-bc73-1a1df84b2d42 /tidb/cdc/default/default/changefeed/info/changefeed-test1 {"upstream-id":7374682541054098141,"namespace":"default","changefeed-id":"changefeed-test1","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:55:08.588312394+08:00","start-ts":450114907929313290,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114907942158350} /tidb/cdc/default/default/changefeed/info/changefeed-test2 {"upstream-id":7374682541054098141,"namespace":"default","changefeed-id":"changefeed-test2","sink-uri":"blackhole://","create-time":"2024-05-30T14:55:08.699975479+08:00","start-ts":450114907968372745,"target-ts":0,"admin-job-type":1,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":32},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"stopped","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114910065786886} /tidb/cdc/default/default/changefeed/info/changefeed-test3 {"upstream-id":7374682541054098141,"namespace":"default","changefeed-id":"changefeed-test3","sink-uri":"blackhole://","create-time":"2024-05-30T14:55:08.804912541+08:00","start-ts":450114907994587147,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114908007694339} /tidb/cdc/default/default/changefeed/status/changefeed-test1 {"checkpoint-ts":450114908125659165,"min-table-barrier-ts":450114911756353544,"admin-job-type":0} /tidb/cdc/default/default/changefeed/status/changefeed-test2 {"checkpoint-ts":450114908125659165,"min-table-barrier-ts":450114909921345542,"admin-job-type":1} /tidb/cdc/default/default/changefeed/status/changefeed-test3 {"checkpoint-ts":450114908125659165,"min-table-barrier-ts":450114911756353544,"admin-job-type":0} /tidb/cdc/default/default/task/position/2762a262-6ae4-45f3-99ca-a7909c3e59d5/changefeed-test1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/2762a262-6ae4-45f3-99ca-a7909c3e59d5/changefeed-test3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/2be5bc53-8a4d-4299-bc73-1a1df84b2d42/changefeed-test1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/2be5bc53-8a4d-4299-bc73-1a1df84b2d42/changefeed-test3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374682541054098141 {"id":7374682541054098141,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list 2>&1 | grep '"address": "127.0.0.1:8300"' wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 14:55:25 CST 2024] <<<<<< run test case foreign_key success! >>>>>> "address": "127.0.0.1:8300", run task successfully pass test: list changefeed pass test: get changefeed wait process cdc.test exit for 1-th time... pass test: pause changefeed wait process cdc.test exit for 2-th time... pass test: resume changefeed wait process cdc.test exit for 3-th time... pass test: rebalance table VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2120d0e40010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-th7j8, pid:6592, start at 2024-05-30 14:55:26.282819388 +0800 CST m=+5.128842908 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:57:26.289 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:55:26.265 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:45:26.265 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2120d0e40010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-th7j8, pid:6592, start at 2024-05-30 14:55:26.282819388 +0800 CST m=+5.128842908 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:57:26.289 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:55:26.265 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:45:26.265 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2120d0c40013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-th7j8, pid:6669, start at 2024-05-30 14:55:26.286999735 +0800 CST m=+5.087361952 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:57:26.294 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:55:26.257 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:45:26.257 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/syncpoint/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/syncpoint/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/syncpoint/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/syncpoint/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/syncpoint/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 14:55:27 CST 2024] <<<<<< START cdc server in consistent_replicate_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=return(true);github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql/MySQLSinkExecDDLDelay=return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_ddl.28112813.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_ddl/cdcconsistent_replicate_ddl.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_ddl/cdc_dataconsistent_replicate_ddl.server2 --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 pass test: move table + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.syncpoint.cli.7933.out cli tso query --pd=http://127.0.0.1:2379 pass test: get processors pass test: list processors Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc) 3309751296 bytes in 6.50 secs (509462790 bytes/sec) [Pipeline] { [Pipeline] cache pass test: set log level + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:55:30 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/9a148d9d-8b5a-4bb2-ba1f-89b3a8c6b05b {UpstreamID:7374682542777058856 Namespace:default ID:9a148d9d-8b5a-4bb2-ba1f-89b3a8c6b05b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:55:00.033579466 +0800 CST StartTs:450114905676185610 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0038a8e10 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114905702400006} {CheckpointTs:450114911797248006 MinTableBarrierTs:450114913645363202 AdminJobType:noop} span: {table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}, resolvedTs: 450114911797248006, checkpointTs: 450114911797248006, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114911797248006, checkpointTs: 450114911797248006, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114911797248006, checkpointTs: 450114911797248006, state: Replicating span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 450114911797248006, checkpointTs: 450114911797248006, state: Replicating span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 450114911797248006, checkpointTs: 450114911797248006, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/38c3f103-b877-4702-b243-9bb250442043 {"id":"38c3f103-b877-4702-b243-9bb250442043","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8478428e7 38c3f103-b877-4702-b243-9bb250442043 /tidb/cdc/default/default/changefeed/info/9a148d9d-8b5a-4bb2-ba1f-89b3a8c6b05b {"upstream-id":7374682542777058856,"namespace":"default","changefeed-id":"9a148d9d-8b5a-4bb2-ba1f-89b3a8c6b05b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:55:00.033579466+08:00","start-ts":450114905676185610,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_ddl/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114905702400006} /tidb/cdc/default/default/changefeed/status/9a148d9d-8b5a-4bb2-ba1f-89b3a8c6b05b {"checkpoint-ts":450114911797248006,"min-table-barrier-ts":450114913645363202,"admin-job-type":0} /tidb/cdc/default/default/task/position/38c3f103-b877-4702-b243-9bb250442043/9a148d9d-8b5a-4bb2-ba1f-89b3a8c6b05b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374682542777058856 {"id":7374682542777058856,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/9a148d9d-8b5a-4bb2-ba1f-89b3a8c6b05b {UpstreamID:7374682542777058856 Namespace:default ID:9a148d9d-8b5a-4bb2-ba1f-89b3a8c6b05b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:55:00.033579466 +0800 CST StartTs:450114905676185610 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0038a8e10 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114905702400006} {CheckpointTs:450114911797248006 MinTableBarrierTs:450114913645363202 AdminJobType:noop} span: {table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}, resolvedTs: 450114911797248006, checkpointTs: 450114911797248006, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114911797248006, checkpointTs: 450114911797248006, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114911797248006, checkpointTs: 450114911797248006, state: Replicating span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 450114911797248006, checkpointTs: 450114911797248006, state: Replicating span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 450114911797248006, checkpointTs: 450114911797248006, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/38c3f103-b877-4702-b243-9bb250442043 {"id":"38c3f103-b877-4702-b243-9bb250442043","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8478428e7 38c3f103-b877-4702-b243-9bb250442043 /tidb/cdc/default/default/changefeed/info/9a148d9d-8b5a-4bb2-ba1f-89b3a8c6b05b {"upstream-id":7374682542777058856,"namespace":"default","changefeed-id":"9a148d9d-8b5a-4bb2-ba1f-89b3a8c6b05b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:55:00.033579466+08:00","start-ts":450114905676185610,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_ddl/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114905702400006} /tidb/cdc/default/default/changefeed/status/9a148d9d-8b5a-4bb2-ba1f-89b3a8c6b05b {"checkpoint-ts":450114911797248006,"min-table-barrier-ts":450114913645363202,"admin-job-type":0} /tidb/cdc/default/default/task/position/38c3f103-b877-4702-b243-9bb250442043/9a148d9d-8b5a-4bb2-ba1f-89b3a8c6b05b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374682542777058856 {"id":7374682542777058856,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/9a148d9d-8b5a-4bb2-ba1f-89b3a8c6b05b {UpstreamID:7374682542777058856 Namespace:default ID:9a148d9d-8b5a-4bb2-ba1f-89b3a8c6b05b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:55:00.033579466 +0800 CST StartTs:450114905676185610 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0038a8e10 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114905702400006} {CheckpointTs:450114911797248006 MinTableBarrierTs:450114913645363202 AdminJobType:noop} span: {table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}, resolvedTs: 450114911797248006, checkpointTs: 450114911797248006, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114911797248006, checkpointTs: 450114911797248006, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114911797248006, checkpointTs: 450114911797248006, state: Replicating span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 450114911797248006, checkpointTs: 450114911797248006, state: Replicating span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 450114911797248006, checkpointTs: 450114911797248006, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/38c3f103-b877-4702-b243-9bb250442043 {"id":"38c3f103-b877-4702-b243-9bb250442043","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8478428e7 38c3f103-b877-4702-b243-9bb250442043 /tidb/cdc/default/default/changefeed/info/9a148d9d-8b5a-4bb2-ba1f-89b3a8c6b05b {"upstream-id":7374682542777058856,"namespace":"default","changefeed-id":"9a148d9d-8b5a-4bb2-ba1f-89b3a8c6b05b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:55:00.033579466+08:00","start-ts":450114905676185610,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_ddl/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":4501149+ grep -q 'etcd info' 05702400006} /tidb/cdc/default/default/changefeed/status/9a148d9d-8b5a-4bb2-ba1f-89b3a8c6b05b {"checkpoint-ts":450114911797248006,"min-table-barrier-ts":450114913645363202,"admin-job-type":0} /tidb/cdc/default/default/task/position/38c3f103-b877-4702-b243-9bb250442043/9a148d9d-8b5a-4bb2-ba1f-89b3a8c6b05b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374682542777058856 {"id":7374682542777058856,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x pass test: remove changefeed pass test: resign owner + set +x + tso='450114913662402562 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114913662402562 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 14:55:31 CST 2024] <<<<<< START cdc server in syncpoint case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/DDLSinkInitializeSlowly=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.syncpoint.79667968.out server --log-file /tmp/tidb_cdc_test/syncpoint/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/syncpoint/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/ddl_puller_lag/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:55:34 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e5798a69-d2f2-4ba4-874d-164ab6b19d0b {"id":"e5798a69-d2f2-4ba4-874d-164ab6b19d0b","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8480c5bce e5798a69-d2f2-4ba4-874d-164ab6b19d0b /tidb/cdc/default/default/upstream/7374682693753175017 {"id":7374682693753175017,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e5798a69-d2f2-4ba4-874d-164ab6b19d0b {"id":"e5798a69-d2f2-4ba4-874d-164ab6b19d0b","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8480c5bce e5798a69-d2f2-4ba4-874d-164ab6b19d0b /tidb/cdc/default/default/upstream/7374682693753175017 {"id":7374682693753175017,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e5798a69-d2f2-4ba4-874d-164ab6b19d0b {"id":"e5798a69-d2f2-4ba4-874d-164ab6b19d0b","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8480c5bce e5798a69-d2f2-4ba4-874d-164ab6b19d0b /tidb/cdc/default/default/upstream/7374682693753175017 {"id":7374682693753175017,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.syncpoint.cli.8017.out cli changefeed create --start-ts=450114913662402562 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/syncpoint/conf/changefeed.toml Create changefeed successfully! ID: 0c27c09d-7cb2-4cb7-919b-2606751f5939 Info: {"upstream_id":7374682693753175017,"namespace":"default","id":"0c27c09d-7cb2-4cb7-919b-2606751f5939","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T14:55:35.28450983+08:00","start_ts":450114913662402562,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":true,"bdr_mode":false,"sync_point_interval":30000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114913662402562,"checkpoint_ts":450114913662402562,"checkpoint_time":"2024-05-30 14:55:30.365"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x ***************** properties ***************** "mysql.user"="root" "recordcount"="10" "workload"="core" "threadcount"="2" "mysql.host"="127.0.0.1" "mysql.port"="4000" "insertproportion"="0" "readallfields"="true" "mysql.db"="testSync" "dotransactions"="false" "updateproportion"="0" "requestdistribution"="uniform" "scanproportion"="0" "operationcount"="0" "readproportion"="0" ********************************************** Run finished, takes 9.77792ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2116.0, Avg(us): 1878, Min(us): 961, Max(us): 5028, 95th(us): 6000, 99th(us): 6000 start tidb cluster in /tmp/tidb_cdc_test/ddl_puller_lag Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc) 3309751296 bytes in 5.35 secs (618668116 bytes/sec) [Pipeline] { [Pipeline] sh [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G03 Run cases: row_format drop_many_tables processor_stop_delay partition_table PROW_JOB_ID=c548ef96-eb30-4461-bb19-187e41d862e3 JENKINS_NODE_COOKIE=8204cae3-c0b7-43d2-9e4e-e74664defc35 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/ GOLANG_VERSION=1.21.10 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-mzktr HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1796069089731416066","prowjobid":"c548ef96-eb30-4461-bb19-187e41d862e3","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#590 TEST_GROUP=G03 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089731416066 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-np16g GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-np16g ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-mzktr GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-mzktr GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=590 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/row_format/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G07 Run cases: kv_client_stream_reconnect cdc split_region PROW_JOB_ID=c548ef96-eb30-4461-bb19-187e41d862e3 JENKINS_NODE_COOKIE=615029bc-3286-4258-8321-87c7b54d197c BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/ GOLANG_VERSION=1.21.10 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l00sh HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1796069089731416066","prowjobid":"c548ef96-eb30-4461-bb19-187e41d862e3","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#590 TEST_GROUP=G07 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089731416066 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-w100l GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-w100l ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l00sh GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l00sh GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=590 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kv_client_stream_reconnect/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G06 Run cases: sink_retry changefeed_error ddl_sequence resourcecontrol PROW_JOB_ID=c548ef96-eb30-4461-bb19-187e41d862e3 JENKINS_NODE_COOKIE=44742ff3-a30a-4e51-8c81-0fde4bb83d8d BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/ GOLANG_VERSION=1.21.10 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t5729 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1796069089731416066","prowjobid":"c548ef96-eb30-4461-bb19-187e41d862e3","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#590 TEST_GROUP=G06 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089731416066 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-1d1v8 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t5729 ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-1d1v8 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t5729 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=590 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/sink_retry/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G08 Run cases: processor_err_chan changefeed_reconstruct multi_capture PROW_JOB_ID=c548ef96-eb30-4461-bb19-187e41d862e3 JENKINS_NODE_COOKIE=a4fa26cb-87e0-4009-900a-31ab242f7ba6 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/ GOLANG_VERSION=1.21.10 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-htrwd HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1796069089731416066","prowjobid":"c548ef96-eb30-4461-bb19-187e41d862e3","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#590 TEST_GROUP=G08 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089731416066 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-6sc1k GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-6sc1k ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-htrwd GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-htrwd GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=590 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/processor_err_chan/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G05 Run cases: charset_gbk ddl_manager multi_source PROW_JOB_ID=c548ef96-eb30-4461-bb19-187e41d862e3 JENKINS_NODE_COOKIE=f798d60f-d997-40a9-ab99-e74db8e54c1d BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/ GOLANG_VERSION=1.21.10 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4168z HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1796069089731416066","prowjobid":"c548ef96-eb30-4461-bb19-187e41d862e3","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#590 TEST_GROUP=G05 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089731416066 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-jw85k GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4168z ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-jw85k GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4168z GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=590 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/charset_gbk/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/row_format Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/kv_client_stream_reconnect Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/sink_retry Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/charset_gbk Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { Verifying downstream PD is started... [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] container [Pipeline] { Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { start tidb cluster in /tmp/tidb_cdc_test/processor_err_chan Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21221eb8000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6w171, pid:3742, start at 2024-05-30 14:55:47.642537038 +0800 CST m=+5.170672489 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:57:47.650 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:55:47.630 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:45:47.630 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21221eb8000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6w171, pid:3742, start at 2024-05-30 14:55:47.642537038 +0800 CST m=+5.170672489 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:57:47.650 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:55:47.630 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:45:47.630 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21221e940014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6w171, pid:3818, start at 2024-05-30 14:55:47.65356658 +0800 CST m=+5.111573618 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:57:47.660 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:55:47.621 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:45:47.621 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] { [Pipeline] { Verifying downstream PD is started... [Pipeline] {Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] cache Starting Upstream TiDB... Verifying downstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.cli.5056.out cli tso query --pd=http://127.0.0.1:2379 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + set +x + tso='450114919106084865 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114919106084865 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Thu May 30 14:55:53 CST 2024] <<<<<< START cdc server in ddl_puller_lag case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorDDLResolved=1*sleep(180000)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.50955097.out server --log-file /tmp/tidb_cdc_test/ddl_puller_lag/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_puller_lag/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21227d940013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t5729, pid:1193, start at 2024-05-30 14:55:53.716320278 +0800 CST m=+5.183752151 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:57:53.724 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:55:53.701 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:45:53.701 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21227d940013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t5729, pid:1193, start at 2024-05-30 14:55:53.716320278 +0800 CST m=+5.183752151 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:57:53.724 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:55:53.701 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:45:53.701 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21227d880013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t5729, pid:1285, start at 2024-05-30 14:55:53.721011124 +0800 CST m=+5.130838003 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:57:53.728 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:55:53.698 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:45:53.698 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sink_retry/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sink_retry/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212284380017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-mzktr, pid:1254, start at 2024-05-30 14:55:54.154504987 +0800 CST m=+5.182398614 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:57:54.162 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:55:54.126 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:45:54.126 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212284380017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-mzktr, pid:1254, start at 2024-05-30 14:55:54.154504987 +0800 CST m=+5.182398614 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:57:54.162 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:55:54.126 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:45:54.126 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212284fc0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-mzktr, pid:1327, start at 2024-05-30 14:55:54.195575719 +0800 CST m=+5.152764854 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:57:54.202 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:55:54.175 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:45:54.175 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/row_format/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/row_format/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/row_format/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/row_format/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/row_format/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc) 3309751296 bytes in 5.36 secs (617449489 bytes/sec) [Pipeline] { + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.cli.2579.out cli tso query --pd=http://127.0.0.1:2379 [Pipeline] cache + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:55:56 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ed7cc630-6ff7-45cb-b5c2-9ecd8820b66a {"id":"ed7cc630-6ff7-45cb-b5c2-9ecd8820b66a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8485f9fdc ed7cc630-6ff7-45cb-b5c2-9ecd8820b66a /tidb/cdc/default/default/upstream/7374682783101676945 {"id":7374682783101676945,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ed7cc630-6ff7-45cb-b5c2-9ecd8820b66a {"id":"ed7cc630-6ff7-45cb-b5c2-9ecd8820b66a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8485f9fdc ed7cc630-6ff7-45cb-b5c2-9ecd8820b66a /tidb/cdc/default/default/upstream/7374682783101676945 {"id":7374682783101676945,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ed7cc630-6ff7-45cb-b5c2-9ecd8820b66a {"id":"ed7cc630-6ff7-45cb-b5c2-9ecd8820b66a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8485f9fdc ed7cc630-6ff7-45cb-b5c2-9ecd8820b66a /tidb/cdc/default/default/upstream/7374682783101676945 {"id":7374682783101676945,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.cli.5143.out cli changefeed create --start-ts=450114919106084865 --sink-uri=mysql+ssl://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: 119742dc-d9eb-4729-b39c-64c0e7b8d076 Info: {"upstream_id":7374682783101676945,"namespace":"default","id":"119742dc-d9eb-4729-b39c-64c0e7b8d076","sink_uri":"mysql+ssl://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T14:55:57.301041999+08:00","start_ts":450114919106084865,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114919106084865,"checkpoint_ts":450114919106084865,"checkpoint_time":"2024-05-30 14:55:51.131"} PASS + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.cli.2575.out cli tso query --pd=http://127.0.0.1:2379 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) coverage: 2.4% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + tso='450114920710406145 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114920710406145 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + set +x + set +x + tso='450114920808710145 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114920808710145 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 14:55:59 CST 2024] <<<<<< START cdc server in row_format case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.26102612.out server --log-file /tmp/tidb_cdc_test/row_format/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/row_format/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ***************** properties ***************** "requestdistribution"="uniform" "insertproportion"="0" "readproportion"="0" "updateproportion"="0" "mysql.host"="127.0.0.1" "mysql.user"="root" "workload"="core" "mysql.port"="4000" "scanproportion"="0" "threadcount"="2" "operationcount"="0" "recordcount"="10" "readallfields"="true" "mysql.db"="sink_retry" "dotransactions"="false" ********************************************** Run finished, takes 11.043439ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2175.5, Avg(us): 2141, Min(us): 905, Max(us): 6378, 95th(us): 7000, 99th(us): 7000 [Thu May 30 14:56:00 CST 2024] <<<<<< START cdc server in sink_retry case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkTxnRandomError=25%return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.26422644.out server --log-file /tmp/tidb_cdc_test/sink_retry/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sink_retry/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2122c3040013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-htrwd, pid:1198, start at 2024-05-30 14:55:58.168563266 +0800 CST m=+5.125006516 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:57:58.177 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:55:58.145 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:45:58.145 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2122c3040013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-htrwd, pid:1198, start at 2024-05-30 14:55:58.168563266 +0800 CST m=+5.125006516 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:57:58.177 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:55:58.145 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:45:58.145 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2122c3f40014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-htrwd, pid:1278, start at 2024-05-30 14:55:58.246590121 +0800 CST m=+5.118433722 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:57:58.254 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:55:58.255 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:45:58.255 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_err_chan/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_err_chan/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:56:02 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5250410b-db7c-4c02-8816-b3e5f784ef79 {"id":"5250410b-db7c-4c02-8816-b3e5f784ef79","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8487516bf 5250410b-db7c-4c02-8816-b3e5f784ef79 /tidb/cdc/default/default/upstream/7374682808665929043 {"id":7374682808665929043,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5250410b-db7c-4c02-8816-b3e5f784ef79 {"id":"5250410b-db7c-4c02-8816-b3e5f784ef79","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8487516bf 5250410b-db7c-4c02-8816-b3e5f784ef79 /tidb/cdc/default/default/upstream/7374682808665929043 {"id":7374682808665929043,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5250410b-db7c-4c02-8816-b3e5f784ef79 {"id":"5250410b-db7c-4c02-8816-b3e5f784ef79","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8487516bf 5250410b-db7c-4c02-8816-b3e5f784ef79 /tidb/cdc/default/default/upstream/7374682808665929043 {"id":7374682808665929043,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.cli.2657.out cli changefeed create --start-ts=450114920808710145 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: dedc6191-d540-4715-bd76-df70faf7cda9 Info: {"upstream_id":7374682808665929043,"namespace":"default","id":"dedc6191-d540-4715-bd76-df70faf7cda9","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T14:56:02.565706927+08:00","start_ts":450114920808710145,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114920808710145,"checkpoint_ts":450114920808710145,"checkpoint_time":"2024-05-30 14:55:57.626"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) pass test: get tso wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21230e100004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l00sh, pid:1233, start at 2024-05-30 14:56:02.953189411 +0800 CST m=+11.847700890 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:02.962 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:02.948 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:02.948 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:56:03 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a2bc1942-6b17-400a-9d91-dc4dc0872d26 {"id":"a2bc1942-6b17-400a-9d91-dc4dc0872d26","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8487725e1 a2bc1942-6b17-400a-9d91-dc4dc0872d26 /tidb/cdc/default/default/upstream/7374682803433309934 {"id":7374682803433309934,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a2bc1942-6b17-400a-9d91-dc4dc0872d26 {"id":"a2bc1942-6b17-400a-9d91-dc4dc0872d26","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8487725e1 a2bc1942-6b17-400a-9d91-dc4dc0872d26 /tidb/cdc/default/default/upstream/7374682803433309934 {"id":7374682803433309934,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a2bc1942-6b17-400a-9d91-dc4dc0872d26 {"id":"a2bc1942-6b17-400a-9d91-dc4dc0872d26","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8487725e1 a2bc1942-6b17-400a-9d91-dc4dc0872d26 /tidb/cdc/default/default/upstream/7374682803433309934 {"id":7374682803433309934,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.cli.2688.out cli changefeed create --start-ts=450114920710406145 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' [Thu May 30 14:56:03 CST 2024] <<<<<< run test case http_api success! >>>>>> + set +x Create changefeed successfully! ID: 983253a1-da7c-486d-9821-758a81f66c2f Info: {"upstream_id":7374682803433309934,"namespace":"default","id":"983253a1-da7c-486d-9821-758a81f66c2f","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T14:56:04.063220449+08:00","start_ts":450114920710406145,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114920710406145,"checkpoint_ts":450114920710406145,"checkpoint_time":"2024-05-30 14:55:57.251"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21230e100004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l00sh, pid:1233, start at 2024-05-30 14:56:02.953189411 +0800 CST m=+11.847700890 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:02.962 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:02.948 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:02.948 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212306300014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l00sh, pid:1310, start at 2024-05-30 14:56:02.485347117 +0800 CST m=+11.273621850 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:02.495 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:02.494 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:02.494 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212326880012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4168z, pid:1235, start at 2024-05-30 14:56:04.562638979 +0800 CST m=+13.491938085 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:04.573 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:04.564 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:04.564 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212326880012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4168z, pid:1235, start at 2024-05-30 14:56:04.562638979 +0800 CST m=+13.491938085 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:04.573 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:04.564 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:04.564 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2123242c0010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4168z, pid:1303, start at 2024-05-30 14:56:04.383897882 +0800 CST m=+13.221986433 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:04.395 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:04.363 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:04.363 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/charset_gbk/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/charset_gbk/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Thu May 30 14:56:07 CST 2024] <<<<<< START cdc server in kv_client_stream_reconnect case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/kv/kvClientForceReconnect=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kv_client_stream_reconnect.25712573.out server --log-file /tmp/tidb_cdc_test/kv_client_stream_reconnect/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kv_client_stream_reconnect/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Thu May 30 14:56:09 CST 2024] <<<<<< START cdc server in charset_gbk case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.charset_gbk.26042606.out server --log-file /tmp/tidb_cdc_test/charset_gbk/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/charset_gbk/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > [Thu May 30 14:56:09 CST 2024] <<<<<< START cdc server in processor_err_chan case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/sourcemanager/puller/ProcessorAddTableError=1*return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_err_chan.27302732.out server --log-file /tmp/tidb_cdc_test/processor_err_chan/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_err_chan/cdc_data --cluster-id default --config /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/processor_err_chan/conf/server.toml --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:56:10 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/785072bb-994a-4b68-b1cc-25f066b3ad7f {"id":"785072bb-994a-4b68-b1cc-25f066b3ad7f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc848778ddb 785072bb-994a-4b68-b1cc-25f066b3ad7f /tidb/cdc/default/default/upstream/7374682807497521357 {"id":7374682807497521357,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/785072bb-994a-4b68-b1cc-25f066b3ad7f {"id":"785072bb-994a-4b68-b1cc-25f066b3ad7f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc848778ddb 785072bb-994a-4b68-b1cc-25f066b3ad7f /tidb/cdc/default/default/upstream/7374682807497521357 {"id":7374682807497521357,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/785072bb-994a-4b68-b1cc-25f066b3ad7f {"id":"785072bb-994a-4b68-b1cc-25f066b3ad7f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc848778ddb 785072bb-994a-4b68-b1cc-25f066b3ad7f /tidb/cdc/default/default/upstream/7374682807497521357 {"id":7374682807497521357,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:56:12 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a311b01f-6223-469e-a003-08173200ded6 {"id":"a311b01f-6223-469e-a003-08173200ded6","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84886c06d a311b01f-6223-469e-a003-08173200ded6 /tidb/cdc/default/default/upstream/7374682822361888896 {"id":7374682822361888896,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a311b01f-6223-469e-a003-08173200ded6 {"id":"a311b01f-6223-469e-a003-08173200ded6","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84886c06d a311b01f-6223-469e-a003-08173200ded6 /tidb/cdc/default/default/upstream/7374682822361888896 {"id":7374682822361888896,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a311b01f-6223-469e-a003-08173200ded6 {"id":"a311b01f-6223-469e-a003-08173200ded6","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84886c06d a311b01f-6223-469e-a003-08173200ded6 /tidb/cdc/default/default/upstream/7374682822361888896 {"id":7374682822361888896,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_changefeed_state http://127.0.0.1:2379 79b55202-b1b5-4f9f-b476-0349e220ad3d normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=79b55202-b1b5-4f9f-b476-0349e220ad3d + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 79b55202-b1b5-4f9f-b476-0349e220ad3d -s + info='{ "upstream_id": 7374682822361888896, "namespace": "default", "id": "79b55202-b1b5-4f9f-b476-0349e220ad3d", "state": "normal", "checkpoint_tso": 450114924785172482, "checkpoint_time": "2024-05-30 14:56:12.795", "error": null }' + echo '{ "upstream_id": 7374682822361888896, "namespace": "default", "id": "79b55202-b1b5-4f9f-b476-0349e220ad3d", "state": "normal", "checkpoint_tso": 450114924785172482, "checkpoint_time": "2024-05-30 14:56:12.795", "error": null }' { "upstream_id": 7374682822361888896, "namespace": "default", "id": "79b55202-b1b5-4f9f-b476-0349e220ad3d", "state": "normal", "checkpoint_tso": 450114924785172482, "checkpoint_time": "2024-05-30 14:56:12.795", "error": null } ++ echo '{' '"upstream_id":' 7374682822361888896, '"namespace":' '"default",' '"id":' '"79b55202-b1b5-4f9f-b476-0349e220ad3d",' '"state":' '"normal",' '"checkpoint_tso":' 450114924785172482, '"checkpoint_time":' '"2024-05-30' '14:56:12.795",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374682822361888896, '"namespace":' '"default",' '"id":' '"79b55202-b1b5-4f9f-b476-0349e220ad3d",' '"state":' '"normal",' '"checkpoint_tso":' 450114924785172482, '"checkpoint_time":' '"2024-05-30' '14:56:12.795",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check diff failed 1-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/http_api_tls/run.sh using Sink-Type: mysql... <<================= WARNING: Running pip install with root privileges is generally not a good idea. Try `__main__.py install --user` instead. Requirement already up-to-date: requests==2.26.0 in /usr/local/lib/python3.6/site-packages Requirement already up-to-date: charset-normalizer~=2.0.0; python_version >= "3" in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) Requirement already up-to-date: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) table row_format.finish_mark not exists for 1-th check, retry later Requirement already up-to-date: certifi>=2017.4.17 in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) Requirement already up-to-date: idna<4,>=2.5; python_version >= "3" in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) The 1 times to try to start tidb cluster... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:56:13 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ec9f4e30-6ae0-4bd8-95d9-4f2c1e1d98b0 {"id":"ec9f4e30-6ae0-4bd8-95d9-4f2c1e1d98b0","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8487d3bd5 ec9f4e30-6ae0-4bd8-95d9-4f2c1e1d98b0 /tidb/cdc/default/default/upstream/7374682820917181213 {"id":7374682820917181213,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ec9f4e30-6ae0-4bd8-95d9-4f2c1e1d98b0 {"id":"ec9f4e30-6ae0-4bd8-95d9-4f2c1e1d98b0","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8487d3bd5 ec9f4e30-6ae0-4bd8-95d9-4f2c1e1d98b0 /tidb/cdc/default/default/upstream/7374682820917181213 {"id":7374682820917181213,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ec9f4e30-6ae0-4bd8-95d9-4f2c1e1d98b0 {"id":"ec9f4e30-6ae0-4bd8-95d9-4f2c1e1d98b0","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8487d3bd5 ec9f4e30-6ae0-4bd8-95d9-4f2c1e1d98b0 /tidb/cdc/default/default/upstream/7374682820917181213 {"id":7374682820917181213,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: 0b6d7676-671f-49f2-a7b3-ed78cb72bad1 Info: {"upstream_id":7374682820917181213,"namespace":"default","id":"0b6d7676-671f-49f2-a7b3-ed78cb72bad1","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T14:56:14.755345572+08:00","start_ts":450114923951554561,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114923951554561,"checkpoint_ts":450114923951554561,"checkpoint_time":"2024-05-30 14:56:09.615"} table row_format.finish_mark not exists for 2-th check, retry later check diff failed 2-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/http_api_tls Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... check diff successfully table row_format.finish_mark not exists for 3-th check, retry later wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 14:56:19 CST 2024] <<<<<< run test case processor_err_chan success! >>>>>> table row_format.finish_mark not exists for 4-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc) 3309751296 bytes in 18.40 secs (179850620 bytes/sec) [Pipeline] { [Pipeline] cache Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table row_format.finish_mark not exists for 5-th check, retry later table charset_gbk_test0.t0 exists table charset_gbk_test0.t1 exists table charset_gbk_test1.t0 not exists for 1-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table row_format.finish_mark not exists for 6-th check, retry later table charset_gbk_test1.t0 exists table test.finish_mark not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table row_format.finish_mark not exists for 7-th check, retry later table test.finish_mark not exists for 2-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table row_format.finish_mark not exists for 8-th check, retry later table test.finish_mark not exists for 3-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table row_format.finish_mark not exists for 9-th check, retry later table test.finish_mark not exists for 4-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table row_format.finish_mark not exists for 10-th check, retry later table test.finish_mark not exists for 5-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc) 3309751296 bytes in 11.41 secs (290111411 bytes/sec) [Pipeline] { [Pipeline] sh =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/changefeed_reconstruct/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh table test.finish_mark exists check table exists success check diff successfully table row_format.finish_mark not exists for 11-th check, retry later + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G09 Run cases: gc_safepoint changefeed_pause_resume cli savepoint synced_status PROW_JOB_ID=c548ef96-eb30-4461-bb19-187e41d862e3 JENKINS_NODE_COOKIE=1bdc23dc-2d57-4e1a-bff9-b856bc011b3f BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/ GOLANG_VERSION=1.21.10 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1796069089731416066","prowjobid":"c548ef96-eb30-4461-bb19-187e41d862e3","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#590 TEST_GROUP=G09 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089731416066 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-6jbwn GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-6jbwn ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=590 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/gc_safepoint/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh wait process cdc.test exit for 1-th time... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G18 Run cases: availability http_proxies sequence PROW_JOB_ID=c548ef96-eb30-4461-bb19-187e41d862e3 JENKINS_NODE_COOKIE=b16a195b-5657-41b1-91d9-d1d8820cf812 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/ GOLANG_VERSION=1.21.10 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-hr7dq HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1796069089731416066","prowjobid":"c548ef96-eb30-4461-bb19-187e41d862e3","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#590 TEST_GROUP=G18 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089731416066 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-vx70x GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-vx70x ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-hr7dq GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-hr7dq GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=590 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/availability/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] cache ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G12 table row_format.finish_mark not exists for 12-th check, retry later wait process cdc.test exit for 4-th time... Run cases: many_pk_or_uk capture_session_done_during_task ddl_attributes PROW_JOB_ID=c548ef96-eb30-4461-bb19-187e41d862e3 JENKINS_NODE_COOKIE=7cb15866-6733-4d29-99ed-a6db4501deb2 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/ GOLANG_VERSION=1.21.10 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-lpxxn HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1796069089731416066","prowjobid":"c548ef96-eb30-4461-bb19-187e41d862e3","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#590 TEST_GROUP=G12 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089731416066 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-wg70k GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-lpxxn ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-wg70k GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-lpxxn GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=590 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/many_pk_or_uk/run.sh using Sink-Type: mysql... <<================= cdc.test: no process found wait process cdc.test exit for 5-th time... process cdc.test already exit [Thu May 30 14:56:37 CST 2024] <<<<<< run test case charset_gbk success! >>>>>> table sink_retry.finish_mark_1 exists check diff successfully ***************** properties ***************** "dotransactions"="false" "workload"="core" "mysql.host"="127.0.0.1" "mysql.user"="root" "recordcount"="10" "operationcount"="0" "updateproportion"="0" "requestdistribution"="uniform" "mysql.db"="sink_retry" "readproportion"="0" "scanproportion"="0" "mysql.port"="4000" "threadcount"="2" "insertproportion"="0" "readallfields"="true" ********************************************** Run finished, takes 3.894734ms INSERT - Takes(s): 0.0, Count: 10, OPS: 3704.6, Avg(us): 665, Min(us): 407, Max(us): 1613, 95th(us): 2000, 99th(us): 2000 start tidb cluster in /tmp/tidb_cdc_test/changefeed_reconstruct Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/gc_safepoint Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... table row_format.finish_mark exists check diff successfully start tidb cluster in /tmp/tidb_cdc_test/availability Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... wait process cdc.test exit for 1-th time... Verifying downstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... The 1 times to try to start tidb cluster... wait process cdc.test exit for 3-th time... wait process cdc.test exit for 4-th time... cdc.test: no process found wait process cdc.test exit for 5-th time... process cdc.test already exit [Thu May 30 14:56:41 CST 2024] <<<<<< run test case row_format success! >>>>>> Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release start tidb cluster in /tmp/tidb_cdc_test/many_pk_or_uk Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ***************** properties ***************** "workload"="core" "insertproportion"="0" "mysql.host"="127.0.0.1" "dotransactions"="false" "mysql.user"="root" "readproportion"="0" "recordcount"="10" "threadcount"="2" "readallfields"="true" "scanproportion"="0" "operationcount"="0" "requestdistribution"="uniform" "mysql.db"="testSync" "mysql.port"="4000" "updateproportion"="0" ********************************************** Run finished, takes 17.548252ms INSERT - Takes(s): 0.0, Count: 10, OPS: 656.6, Avg(us): 3289, Min(us): 1050, Max(us): 8974, 95th(us): 9000, 99th(us): 9000 Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2125925c0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-js3jc, pid:3831, start at 2024-05-30 14:56:44.206125781 +0800 CST m=+20.756005578 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:44.217 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:44.183 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:44.183 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2125925c0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-js3jc, pid:3831, start at 2024-05-30 14:56:44.206125781 +0800 CST m=+20.756005578 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:44.217 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:44.183 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:44.183 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212583c00014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-js3jc, pid:3914, start at 2024-05-30 14:56:43.292416828 +0800 CST m=+19.705083076 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:43.302 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:43.298 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:43.298 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/http_api_tls/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/http_api_tls/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/http_api_tls/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/http_api_tls/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/http_api_tls/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) The 1 times to try to start tls tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/http_api_tls Starting TLS PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 14:56:46 CST 2024] <<<<<< run test case kv_client_stream_reconnect success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/ddl_manager/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2125dbc80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-htrwd, pid:4024, start at 2024-05-30 14:56:48.905163389 +0800 CST m=+5.333165527 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:48.912 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:48.882 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:48.882 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/ddl_manager Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Starting TLS TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/drop_many_tables/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2125dbc80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-htrwd, pid:4024, start at 2024-05-30 14:56:48.905163389 +0800 CST m=+5.333165527 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:48.912 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:48.882 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:48.882 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2125dd380008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-htrwd, pid:4101, start at 2024-05-30 14:56:48.98256062 +0800 CST m=+5.334799738 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:48.992 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:48.974 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:48.974 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2125e85c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:1202, start at 2024-05-30 14:56:49.714704739 +0800 CST m=+5.280564782 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:49.721 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:49.687 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:49.687 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2125e85c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:1202, start at 2024-05-30 14:56:49.714704739 +0800 CST m=+5.280564782 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:49.721 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:49.687 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:49.687 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2125e9cc0015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:1278, start at 2024-05-30 14:56:49.829233897 +0800 CST m=+5.301922930 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:49.837 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:49.830 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:49.830 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/gc_safepoint/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/gc_safepoint/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Verifying downstream PD is started... Starting TLS TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying TLS TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2126156c0019 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-hr7dq, pid:1200, start at 2024-05-30 14:56:52.621342398 +0800 CST m=+7.508704956 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:52.635 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:52.621 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:52.621 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Thu May 30 14:56:53 CST 2024] <<<<<< START cdc server in changefeed_reconstruct case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_reconstruct.53625364.out server --log-file /tmp/tidb_cdc_test/changefeed_reconstruct/cdcserver1.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_reconstruct/cdc_dataserver1 --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 start tidb cluster in /tmp/tidb_cdc_test/drop_many_tables Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21262cd8000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-lpxxn, pid:1235, start at 2024-05-30 14:56:54.091325283 +0800 CST m=+5.569229000 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:54.099 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:54.070 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:54.070 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Thu May 30 14:56:54 CST 2024] <<<<<< START cdc server in gc_safepoint case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/txnutil/gc/InjectGcSafepointUpdateInterval=return(500)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.gc_safepoint.25982600.out server --log-file /tmp/tidb_cdc_test/gc_safepoint/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/gc_safepoint/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2126156c0019 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-hr7dq, pid:1200, start at 2024-05-30 14:56:52.621342398 +0800 CST m=+7.508704956 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:52.635 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:52.621 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:52.621 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212617a80009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-hr7dq, pid:1278, start at 2024-05-30 14:56:52.72577388 +0800 CST m=+7.548981516 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:52.734 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:52.714 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:52.714 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/availability/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/availability/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/availability/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/availability/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/availability/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21262cd8000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-lpxxn, pid:1235, start at 2024-05-30 14:56:54.091325283 +0800 CST m=+5.569229000 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:54.099 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:54.070 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:54.070 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21262f3c0015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-lpxxn, pid:1310, start at 2024-05-30 14:56:54.272488658 +0800 CST m=+5.670106953 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:54.280 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:54.273 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:54.273 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/many_pk_or_uk/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/many_pk_or_uk/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/many_pk_or_uk/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/many_pk_or_uk/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/many_pk_or_uk/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:56:57 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ef36d07e-1927-4ede-b034-b7602da92441 {"id":"ef36d07e-1927-4ede-b034-b7602da92441","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8494c26c2 ef36d07e-1927-4ede-b034-b7602da92441 /tidb/cdc/default/default/upstream/7374683038209590513 {"id":7374683038209590513,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ef36d07e-1927-4ede-b034-b7602da92441 {"id":"ef36d07e-1927-4ede-b034-b7602da92441","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8494c26c2 ef36d07e-1927-4ede-b034-b7602da92441 /tidb/cdc/default/default/upstream/7374683038209590513 {"id":7374683038209590513,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ef36d07e-1927-4ede-b034-b7602da92441 {"id":"ef36d07e-1927-4ede-b034-b7602da92441","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8494c26c2 ef36d07e-1927-4ede-b034-b7602da92441 /tidb/cdc/default/default/upstream/7374683038209590513 {"id":7374683038209590513,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ***************** properties ***************** "scanproportion"="0" "mysql.port"="4000" "dotransactions"="false" "recordcount"="50" "requestdistribution"="uniform" "mysql.db"="changefeed_reconstruct" "updateproportion"="0" "workload"="core" "insertproportion"="0" "operationcount"="0" "readproportion"="0" "threadcount"="4" "mysql.user"="root" "readallfields"="true" "mysql.host"="127.0.0.1" ********************************************** Run finished, takes 18.167489ms INSERT - Takes(s): 0.0, Count: 48, OPS: 3472.1, Avg(us): 1448, Min(us): 992, Max(us): 4328, 95th(us): 5000, 99th(us): 5000 table changefeed_reconstruct.usertable not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:56:57 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/70cdbc97-7036-4ac3-befd-aeec3a77ed4b {"id":"70cdbc97-7036-4ac3-befd-aeec3a77ed4b","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8494dbabb 70cdbc97-7036-4ac3-befd-aeec3a77ed4b /tidb/cdc/default/default/upstream/7374683050581389019 {"id":7374683050581389019,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/70cdbc97-7036-4ac3-befd-aeec3a77ed4b {"id":"70cdbc97-7036-4ac3-befd-aeec3a77ed4b","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8494dbabb 70cdbc97-7036-4ac3-befd-aeec3a77ed4b /tidb/cdc/default/default/upstream/7374683050581389019 {"id":7374683050581389019,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/70cdbc97-7036-4ac3-befd-aeec3a77ed4b {"id":"70cdbc97-7036-4ac3-befd-aeec3a77ed4b","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8494dbabb 70cdbc97-7036-4ac3-befd-aeec3a77ed4b /tidb/cdc/default/default/upstream/7374683050581389019 {"id":7374683050581389019,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x 0 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.cli.2601.out cli tso query --pd=http://127.0.0.1:2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.many_pk_or_uk.cli.2611.out cli tso query --pd=http://127.0.0.1:2379 check diff failed 1-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/cdc/run.sh using Sink-Type: mysql... <<================= table changefeed_reconstruct.usertable exists check diff failed 1-th time, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + tso='450114936732647425 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114936732647425 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x run test case test_kill_owner [Thu May 30 14:57:00 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.26582660.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_kill_owner.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_kill_owner.server1 --cluster-id default + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + tso='450114936994791425 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114936994791425 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 14:57:00 CST 2024] <<<<<< START cdc server in many_pk_or_uk case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.many_pk_or_uk.26422644.out server --log-file /tmp/tidb_cdc_test/many_pk_or_uk/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/many_pk_or_uk/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212673940007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-js3jc, pid:5691, start at 2024-05-30 14:56:58.605031823 +0800 CST m=+6.092724080 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:58.612 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:58.597 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:58.597 +0800 All versions after safe point can be accessed. (DO NOT EDIT) [Thu May 30 14:57:00 CST 2024] <<<<<< START cdc server in http_api_tls case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates ']' + curl_status_cmd='curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client-key.pem -vsL --max-time 20 https://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_api_tls.57505752.out server --log-file /tmp/tidb_cdc_test/http_api_tls/cdc_http_api_tls_tls1.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_api_tls/cdc_data_http_api_tls_tls1 --cluster-id default --config /tmp/tidb_cdc_test/http_api_tls/server.toml --ca /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server-key.pem --cert-allowed-cn client --addr 127.0.0.1:8300 --pd https://127.0.0.1:2579 + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client-key.pem -vsL --max-time 20 https://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 The 1 times to try to start tidb cluster... check diff successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later wait process 5367 exit for 1-th time... wait process 5367 exit for 2-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21269c180011 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4168z, pid:3738, start at 2024-05-30 14:57:01.211294651 +0800 CST m=+5.295841353 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:01.221 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:01.190 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:01.190 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21269c180011 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4168z, pid:3738, start at 2024-05-30 14:57:01.211294651 +0800 CST m=+5.295841353 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:01.221 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:01.190 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:01.190 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21269ccc0015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4168z, pid:3811, start at 2024-05-30 14:57:01.280452462 +0800 CST m=+5.313960561 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:01.287 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:01.285 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:01.285 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_manager/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_manager/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (5367) - No such process wait process 5367 exit for 3-th time... process 5367 already exit check_no_capture http://127.0.0.1:2379 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Thu May 30 14:57:03 CST 2024] <<<<<< START cdc server in changefeed_reconstruct case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_reconstruct.56065608.out server --log-file /tmp/tidb_cdc_test/changefeed_reconstruct/cdcserver2.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_reconstruct/cdc_dataserver2 --cluster-id default --addr 127.0.0.1:8300 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:03 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b6e84775-a822-4e67-b80b-4cddec919c9b {"id":"b6e84775-a822-4e67-b80b-4cddec919c9b","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8495258f3 b6e84775-a822-4e67-b80b-4cddec919c9b /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b6e84775-a822-4e67-b80b-4cddec919c9b {"id":"b6e84775-a822-4e67-b80b-4cddec919c9b","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8495258f3 b6e84775-a822-4e67-b80b-4cddec919c9b /tidb/cdc/default/default/upstream/7374683055714277379 + grep -q 'failed to get info:' {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b6e84775-a822-4e67-b80b-4cddec919c9b {"id":"b6e84775-a822-4e67-b80b-4cddec919c9b","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8495258f3 b6e84775-a822-4e67-b80b-4cddec919c9b /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.cli.2701.out cli changefeed create --start-ts=450114936732647425 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ + (( i++ )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client-key.pem -vsL --max-time 20 https://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Create changefeed successfully! ID: 56d8a316-8373-4096-9cb8-564fb0ca6f6b Info: {"upstream_id":7374683055714277379,"namespace":"default","id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T14:57:03.610004086+08:00","start_ts":450114936732647425,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114936732647425,"checkpoint_ts":450114936732647425,"checkpoint_time":"2024-05-30 14:56:58.371"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... check diff successfully check_safepoint_forward http://127.0.0.1:2379 7374683050581389019 450114937811369988 450114936631721987 * CAfile: /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem CApath: none * NSS: client certificate from file * subject: CN=client * start date: Feb 18 07:48:00 2020 GMT * expire date: Jan 25 07:48:00 2120 GMT * common name: client * issuer: CN=My own CA,O=PingCAP,L=Beijing,ST=Beijing,C=CN * SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 * Server certificate: * subject: CN=tidb-server * start date: Feb 18 09:11:00 2020 GMT * expire date: Jan 25 09:11:00 2120 GMT * common name: tidb-server * issuer: CN=My own CA,O=PingCAP,L=Beijing,ST=Beijing,C=CN > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:03 GMT < Content-Length: 1067 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3f8d25e9-453f-413c-a814-291ebe76df86 {"id":"3f8d25e9-453f-413c-a814-291ebe76df86","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d8fc849740064 3f8d25e9-453f-413c-a814-291ebe76df86 /tidb/cdc/default/default/upstream/7374683086625047040 {"id":7374683086625047040,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3f8d25e9-453f-413c-a814-291ebe76df86 {"id":"3f8d25e9-453f-413c-a814-291ebe76df86","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d8fc849740064 3f8d25e9-453f-413c-a814-291ebe76df86 /tidb/cdc/default/default/upstream/7374683086625047040 {"id":7374683086625047040,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3f8d25e9-453f-413c-a814-291ebe76df86 {"id":"3f8d25e9-453f-413c-a814-291ebe76df86","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d8fc849740064 3f8d25e9-453f-413c-a814-291ebe76df86 /tidb/cdc/default/default/upstream/7374683086625047040 {"id":7374683086625047040,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'etcd info' + break + set +x Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc) 3309751296 bytes in 24.91 secs (132887271 bytes/sec) [Pipeline] { + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:03 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6815e562-7c7b-4e1d-afcb-0c7ab8db2e03 {"id":"6815e562-7c7b-4e1d-afcb-0c7ab8db2e03","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8496163c4 6815e562-7c7b-4e1d-afcb-0c7ab8db2e03 /tidb/cdc/default/default/upstream/7374683064511394726 {"id":7374683064511394726,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6815e562-7c7b-4e1d-afcb-0c7ab8db2e03 {"id":"6815e562-7c7b-4e1d-afcb-0c7ab8db2e03","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8496163c4 6815e562-7c7b-4e1d-afcb-0c7ab8db2e03 /tidb/cdc/default/default/upstream/7374683064511394726 {"id":7374683064511394726,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6815e562-7c7b-4e1d-afcb-0c7ab8db2e03 {"id":"6815e562-7c7b-4e1d-afcb-0c7ab8db2e03","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8496163c4 6815e562-7c7b-4e1d-afcb-0c7ab8db2e03 /tidb/cdc/default/default/upstream/7374683064511394726 {"id":7374683064511394726,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.many_pk_or_uk.cli.2689.out cli changefeed create --start-ts=450114936994791425 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ [Pipeline] sh Create changefeed successfully! ID: c8c97732-3fb0-4eec-bdc4-4aaf110e7a41 Info: {"upstream_id":7374683064511394726,"namespace":"default","id":"c8c97732-3fb0-4eec-bdc4-4aaf110e7a41","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T14:57:04.389251705+08:00","start_ts":450114936994791425,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114936994791425,"checkpoint_ts":450114936994791425,"checkpoint_time":"2024-05-30 14:56:59.371"} PASS [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { coverage: 2.4% of statements in github.com/pingcap/tiflow/... [Pipeline] { [Pipeline] { run task successfully check_changefeed_state http://127.0.0.1:2379 aa240b87-4b39-49f2-923d-b252e1a08675 stopped null + endpoints=http://127.0.0.1:2379 + changefeed_id=aa240b87-4b39-49f2-923d-b252e1a08675 + expected_state=stopped + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c aa240b87-4b39-49f2-923d-b252e1a08675 -s [Thu May 30 14:57:04 CST 2024] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.50635065.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] { [Pipeline] { [Pipeline] { + info='{ "upstream_id": 7374683050581389019, "namespace": "default", "id": "aa240b87-4b39-49f2-923d-b252e1a08675", "state": "stopped", "checkpoint_tso": 450114938086883329, "checkpoint_time": "2024-05-30 14:57:03.537", "error": null }' + echo '{ "upstream_id": 7374683050581389019, "namespace": "default", "id": "aa240b87-4b39-49f2-923d-b252e1a08675", "state": "stopped", "checkpoint_tso": 450114938086883329, "checkpoint_time": "2024-05-30 14:57:03.537", "error": null }' { "upstream_id": 7374683050581389019, "namespace": "default", "id": "aa240b87-4b39-49f2-923d-b252e1a08675", "state": "stopped", "checkpoint_tso": 450114938086883329, "checkpoint_time": "2024-05-30 14:57:03.537", "error": null } ++ echo '{' '"upstream_id":' 7374683050581389019, '"namespace":' '"default",' '"id":' '"aa240b87-4b39-49f2-923d-b252e1a08675",' '"state":' '"stopped",' '"checkpoint_tso":' 450114938086883329, '"checkpoint_time":' '"2024-05-30' '14:57:03.537",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7374683050581389019, '"namespace":' '"default",' '"id":' '"aa240b87-4b39-49f2-923d-b252e1a08675",' '"state":' '"stopped",' '"checkpoint_tso":' 450114938086883329, '"checkpoint_time":' '"2024-05-30' '14:57:03.537",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_equal http://127.0.0.1:2379 7374683050581389019 [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] cache VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2126d7280018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-mzktr, pid:3832, start at 2024-05-30 14:57:05.014682427 +0800 CST m=+5.682301182 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:05.024 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:05.020 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:05.020 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Thu May 30 14:57:05 CST 2024] <<<<<< START cdc server in http_api_tls case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates ']' + GO_FAILPOINTS= + curl_status_cmd='curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client-key.pem -vsL --max-time 20 https://127.0.0.1:8301/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_api_tls.58095811.out server --log-file /tmp/tidb_cdc_test/http_api_tls/cdc_http_api_tls_tls2.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_api_tls/cdc_data_http_api_tls_tls2 --cluster-id default --config /tmp/tidb_cdc_test/http_api_tls/server.toml --ca /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server-key.pem --cert-allowed-cn client --addr 127.0.0.1:8301 --pd https://127.0.0.1:2579 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client-key.pem -vsL --max-time 20 https://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 safepoint is unexpected forward: 450114938073513991 450114936631721987 -> 450114938086883328 450114936631721987 run task failed 1-th time, retry later "is-owner": true, run task successfully + set +x go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/BurntSushi/toml v1.3.2 go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20240527072219-a97e6464c01d go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20240527072219-a97e6464c01d go: downloading github.com/pingcap/failpoint v0.0.0-20240412033321-fd0796e60f86 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:06 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/e33b85ed-fdec-4530-b1e2-36a61d1d1ed2 {UpstreamID:7374683038209590513 Namespace:default ID:e33b85ed-fdec-4530-b1e2-36a61d1d1ed2 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-30 14:56:57.22885398 +0800 CST StartTs:450114936407851009 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0035425a0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114936421220358} {CheckpointTs:450114938636337159 MinTableBarrierTs:450114938636337159 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114938636337159, checkpointTs: 450114938636337159, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c55b5013-381f-4f80-8aa6-f6df28bb66ef {"id":"c55b5013-381f-4f80-8aa6-f6df28bb66ef","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8494c2750 c55b5013-381f-4f80-8aa6-f6df28bb66ef /tidb/cdc/default/default/changefeed/info/e33b85ed-fdec-4530-b1e2-36a61d1d1ed2 {"upstream-id":7374683038209590513,"namespace":"default","changefeed-id":"e33b85ed-fdec-4530-b1e2-36a61d1d1ed2","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T14:56:57.22885398+08:00","start-ts":450114936407851009,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114936421220358} /tidb/cdc/default/default/changefeed/status/e33b85ed-fdec-4530-b1e2-36a61d1d1ed2 {"checkpoint-ts":450114938636337159,"min-table-barrier-ts":450114938636337159,"admin-job-type":0} /tidb/cdc/default/default/task/position/c55b5013-381f-4f80-8aa6-f6df28bb66ef/e33b85ed-fdec-4530-b1e2-36a61d1d1ed2 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683038209590513 {"id":7374683038209590513,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/e33b85ed-fdec-4530-b1e2-36a61d1d1ed2 {UpstreamID:7374683038209590513 Namespace:default ID:e33b85ed-fdec-4530-b1e2-36a61d1d1ed2 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-30 14:56:57.22885398 +0800 CST StartTs:450114936407851009 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0035425a0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114936421220358} {CheckpointTs:450114938636337159 MinTableBarrierTs:450114938636337159 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114938636337159, checkpointTs: 450114938636337159, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c55b5013-381f-4f80-8aa6-f6df28bb66ef {"id":"c55b5013-381f-4f80-8aa6-f6df28bb66ef","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8494c2750 c55b5013-381f-4f80-8aa6-f6df28bb66ef /tidb/cdc/default/default/changefeed/info/e33b85ed-fdec-4530-b1e2-36a61d1d1ed2 {"upstream-id":7374683038209590513,"namespace":"default","changefeed-id":"e33b85ed-fdec-4530-b1e2-36a61d1d1ed2","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T14:56:57.22885398+08:00","start-ts":450114936407851009,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114936421220358} /tidb/cdc/default/default/changefeed/status/e33b85ed-fdec-4530-b1e2-36a61d1d1ed2 {"checkpoint-ts":450114938636337159,"min-table-barrier-ts":450114938636337159,"admin-job-type":0} /tidb/cdc/default/default/task/position/c55b5013-381f-4f80-8aa6-f6df28bb66ef/e33b85ed-fdec-4530-b1e2-36a61d1d1ed2 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683038209590513 {"id":7374683038209590513,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/e33b85ed-fdec-4530-b1e2-36a61d1d1ed2 {UpstreamID:7374683038209590513 Namespace:default ID:e33b85ed-fdec-4530-b1e2-36a61d1d1ed2 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-30 14:56:57.22885398 +0800 CST StartTs:450114936407851009 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0035425a0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114936421220358} {CheckpointTs:450114938636337159 MinTableBarrierTs:450114938636337159 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114938636337159, checkpointTs: 450114938636337159, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c55b5013-381f-4f80-8aa6-f6df28bb66ef {"id":"c55b5013-381f-4f80-8aa6-f6df28bb66ef","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8494c2750 c55b5013-381f-4f80-8aa6-f6df28bb66ef /tidb/cdc/default/default/changefeed/info/e33b85ed-fdec-4530-b1e2-36a61d1d1ed2 {"upstream-id":7374683038209590513,"namespace":"default","changefeed-id":"e33b85ed-fdec-4530-b1e2-36a61d1d1ed2","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T14:56:57.22885398+08:00","start-ts":450114936407851009,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114936421220358} /tidb/cdc/default/default/changefeed/status/e33b85ed-fdec-4530-b1e2-36a61d1d1ed2 {"checkpoint-ts":450114938636337159,"min-table-barrier-ts":450114938636337159,"admin-job-type":0} /tidb/cdc/default/default/task/position/c55b5013-381f-4f80-8aa6-f6df28bb66ef/e33b85ed-fdec-4530-b1e2-36a61d1d1ed2 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683038209590513 {"id":7374683038209590513,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list --pd=http://127.0.0.1:2379 2>&1 | grep id go: downloading go.uber.org/atomic v1.11.0 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading go.uber.org/multierr v1.11.0 + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G15 Run cases: new_ci_collation batch_add_table multi_rocks PROW_JOB_ID=c548ef96-eb30-4461-bb19-187e41d862e3 JENKINS_NODE_COOKIE=d6274049-fc64-4e7f-a49e-f4b3d7fa4d1b BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/ GOLANG_VERSION=1.21.10 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-gszd6 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1796069089731416066","prowjobid":"c548ef96-eb30-4461-bb19-187e41d862e3","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#590 TEST_GROUP=G15 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089731416066 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-df8hn GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-gszd6 ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-df8hn GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-gszd6 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=590 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/new_ci_collation/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2126d7280018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-mzktr, pid:3832, start at 2024-05-30 14:57:05.014682427 +0800 CST m=+5.682301182 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:05.024 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:05.020 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:05.020 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2126d8b40014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-mzktr, pid:3907, start at 2024-05-30 14:57:05.117350564 +0800 CST m=+5.688418344 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:05.128 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:05.119 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:05.119 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/drop_many_tables/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/drop_many_tables/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } owner pid: 2663 owner id b6e84775-a822-4e67-b80b-4cddec919c9b [Thu May 30 14:57:07 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.28022804.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_kill_owner.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_kill_owner.server2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:07 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/23047a58-42a3-4b99-a241-233ec0ce7ffd {"id":"23047a58-42a3-4b99-a241-233ec0ce7ffd","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8497e61bb 23047a58-42a3-4b99-a241-233ec0ce7ffd /tidb/cdc/default/default/upstream/7374683091896812572 {"id":7374683091896812572,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/23047a58-42a3-4b99-a241-233ec0ce7ffd {"id":"23047a58-42a3-4b99-a241-233ec0ce7ffd","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8497e61bb 23047a58-42a3-4b99-a241-233ec0ce7ffd /tidb/cdc/default/default/upstream/7374683091896812572 {"id":7374683091896812572,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/23047a58-42a3-4b99-a241-233ec0ce7ffd {"id":"23047a58-42a3-4b99-a241-233ec0ce7ffd","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8497e61bb 23047a58-42a3-4b99-a241-233ec0ce7ffd /tidb/cdc/default/default/upstream/7374683091896812572 {"id":7374683091896812572,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.cli.5117.out cli changefeed create --sink-uri=mysql://root@127.0.0.1:3306/ -c=ddl-manager "id": "c55b5013-381f-4f80-8aa6-f6df28bb66ef", "cluster-id": "default" run task successfully check_safepoint_equal http://127.0.0.1:2379 7374683050581389019 Create changefeed successfully! ID: ddl-manager Info: {"upstream_id":7374683091896812572,"namespace":"default","id":"ddl-manager","sink_uri":"mysql://root@127.0.0.1:3306/","create_time":"2024-05-30T14:57:08.162017979+08:00","start_ts":450114939267579905,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114939267579905,"checkpoint_ts":450114939267579905,"checkpoint_time":"2024-05-30 14:57:08.041"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client-key.pem -vsL --max-time 20 https://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem CApath: none * NSS: client certificate from file * subject: CN=client * start date: Feb 18 07:48:00 2020 GMT * expire date: Jan 25 07:48:00 2120 GMT * common name: client * issuer: CN=My own CA,O=PingCAP,L=Beijing,ST=Beijing,C=CN * SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 * Server certificate: * subject: CN=tidb-server * start date: Feb 18 09:11:00 2020 GMT * expire date: Jan 25 09:11:00 2120 GMT * common name: tidb-server * issuer: CN=My own CA,O=PingCAP,L=Beijing,ST=Beijing,C=CN > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:09 GMT < Content-Length: 1321 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3f8d25e9-453f-413c-a814-291ebe76df86 {"id":"3f8d25e9-453f-413c-a814-291ebe76df86","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/edf375ae-b1ba-4424-8512-c33ec57b25ac {"id":"edf375ae-b1ba-4424-8512-c33ec57b25ac","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d8fc849740064 3f8d25e9-453f-413c-a814-291ebe76df86 /tidb/cdc/default/__cdc_meta__/owner/1f5d8fc849740098 edf375ae-b1ba-4424-8512-c33ec57b25ac /tidb/cdc/default/default/upstream/7374683086625047040 {"id":7374683086625047040,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3f8d25e9-453f-413c-a814-291ebe76df86 {"id":"3f8d25e9-453f-413c-a814-291ebe76df86","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/edf375ae-b1ba-4424-8512-c33ec57b25ac {"id":"edf375ae-b1ba-4424-8512-c33ec57b25ac","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d8fc849740064 3f8d25e9-453f-413c-a814-291ebe76df86 /tidb/cdc/default/__cdc_meta__/owner/1f5d8fc849740098 edf375ae-b1ba-4424-8512-c33ec57b25ac /tidb/cdc/default/default/upstream/7374683086625047040 {"id":7374683086625047040,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3f8d25e9-453f-413c-a814-291ebe76df86 {"id":"3f8d25e9-453f-413c-a814-291ebe76df86","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/edf375ae-b1ba-4424-8512-c33ec57b25ac {"id":"edf375ae-b1ba-4424-8512-c33ec57b25ac","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d8fc849740064 3f8d25e9-453f-413c-a814-291ebe76df86 /tidb/cdc/default/__cdc_meta__/owner/1f5d8fc849740098 edf375ae-b1ba-4424-8512-c33ec57b25ac /tidb/cdc/default/default/upstream/7374683086625047040 {"id":7374683086625047040,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'etcd info' + break + set +x go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading golang.org/x/text v0.15.0 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/pingcap/kvproto v0.0.0-20240522024016-df42997c2c57 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/shirou/gopsutil/v3 v3.24.4 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/golang/protobuf v1.5.4 go: downloading github.com/pingcap/tipb v0.0.0-20240507090649-2bf6bb0cb996 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842 go: downloading golang.org/x/sync v0.7.0 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20240510070134-e7f530c64790 go: downloading github.com/gorilla/mux v1.8.0 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading google.golang.org/grpc v1.64.0 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/danjacques/gofslock v0.0.0-20220131014315-6e321f4509c8 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/tikv/pd/client v0.0.0-20240210135946-3488a653ddd9 go: downloading github.com/prometheus/client_golang v1.19.1 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/google/uuid v1.6.0 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/google/btree v1.1.2 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading github.com/stretchr/testify v1.9.0 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading google.golang.org/protobuf v1.34.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/aws/aws-sdk-go v1.44.259 go: downloading github.com/klauspost/compress v1.17.8 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.6 go: downloading cloud.google.com/go v0.112.2 go: downloading google.golang.org/api v0.170.0 go: downloading golang.org/x/oauth2 v0.18.0 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/pkg/errors v0.9.1 capture_id: c55b5013-381f-4f80-8aa6-f6df28bb66ef check_processor_table_count http://127.0.0.1:2379 e33b85ed-fdec-4530-b1e2-36a61d1d1ed2 c55b5013-381f-4f80-8aa6-f6df28bb66ef 1 go: downloading golang.org/x/sys v0.20.0 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/iancoleman/strcase v0.2.0 go: downloading github.com/prometheus/common v0.53.0 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/cockroachdb/pebble v0.0.0-20220415182917-06c9d3be25b3 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading golang.org/x/time v0.5.0 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 + set +x run task successfully go: downloading golang.org/x/net v0.25.0 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/prometheus/procfs v0.15.0 table sink_retry.finish_mark_2 exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 14:57:07 CST 2024] <<<<<< run test case sink_retry success! >>>>>> go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.1 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.1 go: downloading golang.org/x/crypto v0.23.0 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/xitongsys/parquet-go v1.6.0 go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/sasha-s/go-deadlock v0.3.1 go: downloading github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.cli.5163.out cli tso query --pd=http://127.0.0.1:2379 go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/pingcap/badger v1.5.1-0.20230103063557-828f39b09b6d go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/robfig/cron v1.2.0 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240521202816-d264139d666e go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/fatih/color v1.16.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/mattn/go-runewidth v0.0.15 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading github.com/petermattis/goid v0.0.0-20231207134359-e60b3f734c67 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.12.0 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/golang-jwt/jwt/v5 v5.2.0 go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/DataDog/zstd v1.5.5 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/klauspost/cpuid v1.3.1 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > go: downloading github.com/go-logr/logr v1.4.1 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading go.opentelemetry.io/otel/metric v1.24.0 start tidb cluster in /tmp/tidb_cdc_test/cdc Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:11 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003615680 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114937165185053 MinTableBarrierTs:450114939826208774 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114937165185053, checkpointTs: 450114937165185053, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/06609d24-ddf4-4ef6-a00e-5afe8021ba06 {"id":"06609d24-ddf4-4ef6-a00e-5afe8021ba06","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/b6e84775-a822-4e67-b80b-4cddec919c9b {"id":"b6e84775-a822-4e67-b80b-4cddec919c9b","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8495258f3 b6e84775-a822-4e67-b80b-4cddec919c9b /tidb/cdc/default/__cdc_meta__/owner/22318fc84952594e 06609d24-ddf4-4ef6-a00e-5afe8021ba06 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114937165185053,"min-table-barrier-ts":450114939826208774,"admin-job-type":0} /tidb/cdc/default/default/task/position/06609d24-ddf4-4ef6-a00e-5afe8021ba06/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b6e84775-a822-4e67-b80b-4cddec919c9b/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003615680 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114937165185053 MinTableBarrierTs:450114939826208774 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114937165185053, checkpointTs: 450114937165185053, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/06609d24-ddf4-4ef6-a00e-5afe8021ba06 {"id":"06609d24-ddf4-4ef6-a00e-5afe8021ba06","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/b6e84775-a822-4e67-b80b-4cddec919c9b {"id":"b6e84775-a822-4e67-b80b-4cddec919c9b","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8495258f3 b6e84775-a822-4e67-b80b-4cddec919c9b /tidb/cdc/default/__cdc_meta__/owner/22318fc84952594e 06609d24-ddf4-4ef6-a00e-5afe8021ba06 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114937165185053,"min-table-barrier-ts":450114939826208774,"admin-job-type":0} /tidb/cdc/default/default/task/position/06609d24-ddf4-4ef6-a00e-5afe8021ba06/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b6e84775-a822-4e67-b80b-4cddec919c9b/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003615680 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114937165185053 MinTableBarrierTs:450114939826208774 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114937165185053, checkpointTs: 450114937165185053, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/06609d24-ddf4-4ef6-a00e-5afe8021ba06 {"id":"06609d24-ddf4-4ef6-a00e-5afe8021ba06","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/b6e84775-a822-4e67-b80b-4cddec919c9b {"id":"b6e84775-a822-4e67-b80b-4cddec919c9b","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8495258f3 b6e84775-a822-4e67-b80b-4cddec919c9b /tidb/cdc/default/__cdc_meta__/owner/22318fc84952594e 06609d24-ddf4-4ef6-a00e-5afe8021ba06 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114937165185053,"min-table-barrier-ts":450114939826208774,"admin-job-type":0} /tidb/cdc/default/default/task/position/06609d24-ddf4-4ef6-a00e-5afe8021ba06/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b6e84775-a822-4e67-b80b-4cddec919c9b/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/up+ grep -q 'etcd info' stream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list 2>&1 | grep -v "b6e84775-a822-4e67-b80b-4cddec919c9b" | grep -v cluster_id | grep id + set +x + tso='450114939799732225 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114939799732225 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 14:57:11 CST 2024] <<<<<< START cdc server in drop_many_tables case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.51855187.out server --log-file /tmp/tidb_cdc_test/drop_many_tables/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/drop_many_tables/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 pass test: check health pass test: get status check_processor_table_count http://127.0.0.1:2379 e33b85ed-fdec-4530-b1e2-36a61d1d1ed2 c55b5013-381f-4f80-8aa6-f6df28bb66ef 0 start tidb cluster in /tmp/tidb_cdc_test/new_ci_collation Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... run task successfully pass test: create changefeed check_changefeed_state https://127.0.0.1:2579 changefeed-test1 normal null /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates + endpoints=https://127.0.0.1:2579 + changefeed_id=changefeed-test1 + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c changefeed-test1 -s check_changefeed_state http://127.0.0.1:2379 aa240b87-4b39-49f2-923d-b252e1a08675 normal null + endpoints=http://127.0.0.1:2379 "cluster-id": "default" + changefeed_id=aa240b87-4b39-49f2-923d-b252e1a08675 + expected_state=normal "id": "06609d24-ddf4-4ef6-a00e-5afe8021ba06", "cluster-id": "default" + error_msg=null + tls_dir=null run task successfully + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c aa240b87-4b39-49f2-923d-b252e1a08675 -s table count 1 does equal to expected count 0 run task failed 1-th time, retry later + info='{ "upstream_id": 7374683086625047040, "namespace": "default", "id": "changefeed-test1", "state": "normal", "checkpoint_tso": 450114940226240521, "checkpoint_time": "2024-05-30 14:57:11.698", "error": null }' + echo '{ "upstream_id": 7374683086625047040, "namespace": "default", "id": "changefeed-test1", "state": "normal", "checkpoint_tso": 450114940226240521, "checkpoint_time": "2024-05-30 14:57:11.698", "error": null }' { "upstream_id": 7374683086625047040, "namespace": "default", "id": "changefeed-test1", "state": "normal", "checkpoint_tso": 450114940226240521, "checkpoint_time": "2024-05-30 14:57:11.698", "error": null } ++ echo '{' '"upstream_id":' 7374683086625047040, '"namespace":' '"default",' '"id":' '"changefeed-test1",' '"state":' '"normal",' '"checkpoint_tso":' 450114940226240521, '"checkpoint_time":' '"2024-05-30' '14:57:11.698",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374683086625047040, '"namespace":' '"default",' '"id":' '"changefeed-test1",' '"state":' '"normal",' '"checkpoint_tso":' 450114940226240521, '"checkpoint_time":' '"2024-05-30' '14:57:11.698",' '"error":' null '}' ++ jq -r .error.message + info='{ "upstream_id": 7374683050581389019, "namespace": "default", "id": "aa240b87-4b39-49f2-923d-b252e1a08675", "state": "normal", "checkpoint_tso": 450114938086883329, "checkpoint_time": "2024-05-30 14:57:03.537", "error": null }' + echo '{ "upstream_id": 7374683050581389019, "namespace": "default", "id": "aa240b87-4b39-49f2-923d-b252e1a08675", "state": "normal", "checkpoint_tso": 450114938086883329, "checkpoint_time": "2024-05-30 14:57:03.537", "error": null }' { "upstream_id": 7374683050581389019, "namespace": "default", "id": "aa240b87-4b39-49f2-923d-b252e1a08675", "state": "normal", "checkpoint_tso": 450114938086883329, "checkpoint_time": "2024-05-30 14:57:03.537", "error": null } ++ echo '{' '"upstream_id":' 7374683050581389019, '"namespace":' '"default",' '"id":' '"aa240b87-4b39-49f2-923d-b252e1a08675",' '"state":' '"normal",' '"checkpoint_tso":' 450114938086883329, '"checkpoint_time":' '"2024-05-30' '14:57:03.537",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374683050581389019, '"namespace":' '"default",' '"id":' '"aa240b87-4b39-49f2-923d-b252e1a08675",' '"state":' '"normal",' '"checkpoint_tso":' 450114938086883329, '"checkpoint_time":' '"2024-05-30' '14:57:03.537",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_forward http://127.0.0.1:2379 7374683050581389019 450114938086883328 450114938086883329 + message=null + [[ ! null =~ null ]] run task successfully check_changefeed_state https://127.0.0.1:2579 changefeed-test2 normal null /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates + endpoints=https://127.0.0.1:2579 + changefeed_id=changefeed-test2 + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c changefeed-test2 -s + info='{ "upstream_id": 7374683086625047040, "namespace": "default", "id": "changefeed-test2", "state": "normal", "checkpoint_tso": 450114940291776521, "checkpoint_time": "2024-05-30 14:57:11.948", "error": null }' + echo '{ "upstream_id": 7374683086625047040, "namespace": "default", "id": "changefeed-test2", "state": "normal", "checkpoint_tso": 450114940291776521, "checkpoint_time": "2024-05-30 14:57:11.948", "error": null }' { "upstream_id": 7374683086625047040, "namespace": "default", "id": "changefeed-test2", "state": "normal", "checkpoint_tso": 450114940291776521, "checkpoint_time": "2024-05-30 14:57:11.948", "error": null } ++ echo '{' '"upstream_id":' 7374683086625047040, '"namespace":' '"default",' '"id":' '"changefeed-test2",' '"state":' '"normal",' '"checkpoint_tso":' 450114940291776521, '"checkpoint_time":' '"2024-05-30' '14:57:11.948",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374683086625047040, '"namespace":' '"default",' '"id":' '"changefeed-test2",' '"state":' '"normal",' '"checkpoint_tso":' 450114940291776521, '"checkpoint_time":' '"2024-05-30' '14:57:11.948",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_changefeed_state https://127.0.0.1:2579 changefeed-test3 normal null /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates + endpoints=https://127.0.0.1:2579 + changefeed_id=changefeed-test3 + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c changefeed-test3 -s + info='{ "upstream_id": 7374683086625047040, "namespace": "default", "id": "changefeed-test3", "state": "normal", "checkpoint_tso": 450114940344205323, "checkpoint_time": "2024-05-30 14:57:12.148", "error": null }' + echo '{ "upstream_id": 7374683086625047040, "namespace": "default", "id": "changefeed-test3", "state": "normal", "checkpoint_tso": 450114940344205323, "checkpoint_time": "2024-05-30 14:57:12.148", "error": null }' { "upstream_id": 7374683086625047040, "namespace": "default", "id": "changefeed-test3", "state": "normal", "checkpoint_tso": 450114940344205323, "checkpoint_time": "2024-05-30 14:57:12.148", "error": null } ++ echo '{' '"upstream_id":' 7374683086625047040, '"namespace":' '"default",' '"id":' '"changefeed-test3",' '"state":' '"normal",' '"checkpoint_tso":' 450114940344205323, '"checkpoint_time":' '"2024-05-30' '14:57:12.148",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374683086625047040, '"namespace":' '"default",' '"id":' '"changefeed-test3",' '"state":' '"normal",' '"checkpoint_tso":' 450114940344205323, '"checkpoint_time":' '"2024-05-30' '14:57:12.148",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_changefeed_state https://127.0.0.1:2579 changefeed-test4 normal null /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates + endpoints=https://127.0.0.1:2579 + changefeed_id=changefeed-test4 + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c changefeed-test4 -s Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release wait process 5068 exit for 1-th time... wait process 5068 exit for 2-th time... + info='{ "upstream_id": 7374683086625047040, "namespace": "default", "id": "changefeed-test4", "state": "normal", "checkpoint_tso": 450114940383526920, "checkpoint_time": "2024-05-30 14:57:12.298", "error": null }' + echo '{ "upstream_id": 7374683086625047040, "namespace": "default", "id": "changefeed-test4", "state": "normal", "checkpoint_tso": 450114940383526920, "checkpoint_time": "2024-05-30 14:57:12.298", "error": null }' { "upstream_id": 7374683086625047040, "namespace": "default", "id": "changefeed-test4", "state": "normal", "checkpoint_tso": 450114940383526920, "checkpoint_time": "2024-05-30 14:57:12.298", "error": null } ++ echo '{' '"upstream_id":' 7374683086625047040, '"namespace":' '"default",' '"id":' '"changefeed-test4",' '"state":' '"normal",' '"checkpoint_tso":' 450114940383526920, '"checkpoint_time":' '"2024-05-30' '14:57:12.298",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374683086625047040, '"namespace":' '"default",' '"id":' '"changefeed-test4",' '"state":' '"normal",' '"checkpoint_tso":' 450114940383526920, '"checkpoint_time":' '"2024-05-30' '14:57:12.298",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully capture_id: 06609d24-ddf4-4ef6-a00e-5afe8021ba06 wait process 5068 exit for 3-th time... run task successfully check_changefeed_state http://127.0.0.1:2379 aa240b87-4b39-49f2-923d-b252e1a08675 stopped null + endpoints=http://127.0.0.1:2379 + changefeed_id=aa240b87-4b39-49f2-923d-b252e1a08675 + expected_state=stopped + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c aa240b87-4b39-49f2-923d-b252e1a08675 -s go: downloading github.com/jmespath/go-jmespath v0.4.0 + info='{ "upstream_id": 7374683050581389019, "namespace": "default", "id": "aa240b87-4b39-49f2-923d-b252e1a08675", "state": "stopped", "checkpoint_tso": 450114938086883329, "checkpoint_time": "2024-05-30 14:57:03.537", "error": null }' + echo '{ "upstream_id": 7374683050581389019, "namespace": "default", "id": "aa240b87-4b39-49f2-923d-b252e1a08675", "state": "stopped", "checkpoint_tso": 450114938086883329, "checkpoint_time": "2024-05-30 14:57:03.537", "error": null }' { "upstream_id": 7374683050581389019, "namespace": "default", "id": "aa240b87-4b39-49f2-923d-b252e1a08675", "state": "stopped", "checkpoint_tso": 450114938086883329, "checkpoint_time": "2024-05-30 14:57:03.537", "error": null } ++ echo '{' '"upstream_id":' 7374683050581389019, '"namespace":' '"default",' '"id":' '"aa240b87-4b39-49f2-923d-b252e1a08675",' '"state":' '"stopped",' '"checkpoint_tso":' 450114938086883329, '"checkpoint_time":' '"2024-05-30' '14:57:03.537",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7374683050581389019, '"namespace":' '"default",' '"id":' '"aa240b87-4b39-49f2-923d-b252e1a08675",' '"state":' '"stopped",' '"checkpoint_tso":' 450114938086883329, '"checkpoint_time":' '"2024-05-30' '14:57:03.537",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully wait process 2663 exit for 1-th time... /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (5068) - No such process wait process 5068 exit for 4-th time... process 5068 already exit [Thu May 30 14:57:14 CST 2024] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.51855187.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check_processor_table_count http://127.0.0.1:2379 e33b85ed-fdec-4530-b1e2-36a61d1d1ed2 c55b5013-381f-4f80-8aa6-f6df28bb66ef 0 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading github.com/google/s2a-go v0.1.7 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 check_changefeed_state http://127.0.0.1:2379 4791f1bd-071b-4a01-9acf-03f236d1dc41 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=4791f1bd-071b-4a01-9acf-03f236d1dc41 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 4791f1bd-071b-4a01-9acf-03f236d1dc41 -s + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:14 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e52ea386-d723-4fbc-9668-76113e0afa69 {"id":"e52ea386-d723-4fbc-9668-76113e0afa69","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84989e1c0 e52ea386-d723-4fbc-9668-76113e0afa69 /tidb/cdc/default/default/upstream/7374683117357670806 {"id":7374683117357670806,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e52ea386-d723-4fbc-9668-76113e0afa69 {"id":"e52ea386-d723-4fbc-9668-76113e0afa69","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84989e1c0 e52ea386-d723-4fbc-9668-76113e0afa69 /tidb/cdc/default/default/upstream/7374683117357670806 {"id":7374683117357670806,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e52ea386-d723-4fbc-9668-76113e0afa69 {"id":"e52ea386-d723-4fbc-9668-76113e0afa69","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84989e1c0 e52ea386-d723-4fbc-9668-76113e0afa69 /tidb/cdc/default/default/upstream/7374683117357670806 {"id":7374683117357670806,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.cli.5229.out cli changefeed create --start-ts=450114939799732225 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ run task successfully go: downloading github.com/felixge/httpsnoop v1.0.4 Create changefeed successfully! ID: 778d6352-a826-4fa8-83fc-79fa2812c341 Info: {"upstream_id":7374683117357670806,"namespace":"default","id":"778d6352-a826-4fa8-83fc-79fa2812c341","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T14:57:15.099205562+08:00","start_ts":450114939799732225,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114939799732225,"checkpoint_ts":450114939799732225,"checkpoint_time":"2024-05-30 14:57:10.071"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... wait process 2663 exit for 2-th time... + info='{ "upstream_id": 7374683050581389019, "namespace": "default", "id": "4791f1bd-071b-4a01-9acf-03f236d1dc41", "state": "normal", "checkpoint_tso": 450114940970205189, "checkpoint_time": "2024-05-30 14:57:14.536", "error": null }' + echo '{ "upstream_id": 7374683050581389019, "namespace": "default", "id": "4791f1bd-071b-4a01-9acf-03f236d1dc41", "state": "normal", "checkpoint_tso": 450114940970205189, "checkpoint_time": "2024-05-30 14:57:14.536", "error": null }' { "upstream_id": 7374683050581389019, "namespace": "default", "id": "4791f1bd-071b-4a01-9acf-03f236d1dc41", "state": "normal", "checkpoint_tso": 450114940970205189, "checkpoint_time": "2024-05-30 14:57:14.536", "error": null } ++ echo '{' '"upstream_id":' 7374683050581389019, '"namespace":' '"default",' '"id":' '"4791f1bd-071b-4a01-9acf-03f236d1dc41",' '"state":' '"normal",' '"checkpoint_tso":' 450114940970205189, '"checkpoint_time":' '"2024-05-30' '14:57:14.536",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374683050581389019, '"namespace":' '"default",' '"id":' '"4791f1bd-071b-4a01-9acf-03f236d1dc41",' '"state":' '"normal",' '"checkpoint_tso":' 450114940970205189, '"checkpoint_time":' '"2024-05-30' '14:57:14.536",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_equal http://127.0.0.1:2379 7374683050581389019 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... wait process 2663 exit for 3-th time... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc) 3309751296 bytes in 7.53 secs (439395950 bytes/sec) [Pipeline] { /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (2663) - No such process wait process 2663 exit for 4-th time... process 2663 already exit cdc.test cli capture list --server 'http://127.0.0.1:8301' 2>&1 |grep 06609d24-ddf4-4ef6-a00e-5afe8021ba06 -A1 | grep '"is-owner": true' [Pipeline] cache + set +x ***************** properties ***************** "mysql.port"="4000" "operationcount"="0" "requestdistribution"="uniform" "mysql.host"="127.0.0.1" "mysql.user"="root" "updateproportion"="0" "recordcount"="50" "scanproportion"="0" "readallfields"="true" "workload"="core" "insertproportion"="0" "mysql.db"="changefeed_reconstruct" "threadcount"="4" "dotransactions"="false" "readproportion"="0" ********************************************** table test.simple1 exists pass test: list changefeed Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) "is-owner": true, run task successfully test_kill_owner: pass + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:17 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7374683091896812572 Namespace:default ID:ddl-manager SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:08.162017979 +0800 CST StartTs:450114939267579905 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004138bd0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114939293794313} {CheckpointTs:450114939687010317 MinTableBarrierTs:450114939687010317 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7b7f6885-d525-4e15-b825-3fda4d9000b6 {"id":"7b7f6885-d525-4e15-b825-3fda4d9000b6","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8497e63df 7b7f6885-d525-4e15-b825-3fda4d9000b6 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7374683091896812572,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2024-05-30T14:57:08.162017979+08:00","start-ts":450114939267579905,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114939293794313} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":450114939687010317,"min-table-barrier-ts":450114939687010317,"admin-job-type":0} /tidb/cdc/default/default/task/position/7b7f6885-d525-4e15-b825-3fda4d9000b6/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683091896812572 {"id":7374683091896812572,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7374683091896812572 Namespace:default ID:ddl-manager SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:08.162017979 +0800 CST StartTs:450114939267579905 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004138bd0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114939293794313} {CheckpointTs:450114939687010317 MinTableBarrierTs:450114939687010317 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7b7f6885-d525-4e15-b825-3fda4d9000b6 {"id":"7b7f6885-d525-4e15-b825-3fda4d9000b6","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8497e63df 7b7f6885-d525-4e15-b825-3fda4d9000b6 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7374683091896812572,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2024-05-30T14:57:08.162017979+08:00","start-ts":450114939267579905,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114939293794313} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":450114939687010317,"min-table-barrier-ts":450114939687010317,"admin-job-type":0} /tidb/cdc/default/default/task/position/7b7f6885-d525-4e15-b825-3fda4d9000b6/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683091896812572 {"id":7374683091896812572,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7374683091896812572 Namespace:default ID:ddl-manager SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:08.162017979 +0800 CST StartTs:450114939267579905 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004138bd0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114939293794313} {CheckpointTs:450114939687010317 MinTableBarrierTs:450114939687010317 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7b7f6885-d525-4e15-b825-3fda4d9000b6 {"id":"7b7f6885-d525-4e15-b825-3fda4d9000b6","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8497e63df 7b7f6885-d525-4e15-b825-3fda4d9000b6 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7374683091896812572,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2024-05-30T14:57:08.162017979+08:00","start-ts":450114939267579905,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114939293794313} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":450114939687010317,"min-table-barrier-ts":450114939687010317,"admin-job-type":0} /tidb/cdc/default/default/task/position/7b7f6885-d525-4e15-b825-3fda4d9000b6/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683091896812572 + grep -q 'etcd info' {"id":7374683091896812572,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x [Thu May 30 14:57:17 CST 2024] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.52425244.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > table drop_tables.c not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/changefeed_error/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... wait process cdc.test exit for 1-th time... run task successfully wait process cdc.test exit for 2-th time... Run finished, takes 1.907451928s INSERT - Takes(s): 1.6, Count: 48, OPS: 30.0, Avg(us): 154896, Min(us): 63797, Max(us): 395474, 95th(us): 306000, 99th(us): 396000 table changefeed_reconstruct.usertable exists Changefeed remove successfully. ID: aa240b87-4b39-49f2-923d-b252e1a08675 CheckpointTs: 450114938086883329 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1 check_safepoint_forward http://127.0.0.1:2379 7374683050581389019 450114940970205188 450114940970205189 450114938086883329 wait process cdc.test exit for 3-th time... check diff failed 1-th time, retry later pass test: get changefeed cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit run test case test_hang_up_owner [Thu May 30 14:57:19 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.29762978.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_hang_up_owner.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_hang_up_owner.server1 --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table drop_tables.c not exists for 2-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) run task successfully pass test: pause changefeed Changefeed remove successfully. ID: 4791f1bd-071b-4a01-9acf-03f236d1dc41 CheckpointTs: 450114942267817990 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1 check_safepoint_cleared http://127.0.0.1:2379 7374683050581389019 run task successfully start tidb cluster in /tmp/tidb_cdc_test/changefeed_error Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... wait process cdc.test exit for 1-th time... check diff successfully wait process cdc.test exit for 2-th time... pass test: update changefeed wait process cdc.test exit for 1-th time... table drop_tables.c not exists for 3-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:21 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7374683091896812572 Namespace:default ID:ddl-manager SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:08.162017979 +0800 CST StartTs:450114939267579905 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004138bd0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114939293794313} {CheckpointTs:450114939687010317 MinTableBarrierTs:450114939687010317 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7b7f6885-d525-4e15-b825-3fda4d9000b6 {"id":"7b7f6885-d525-4e15-b825-3fda4d9000b6","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8497e63df 7b7f6885-d525-4e15-b825-3fda4d9000b6 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7374683091896812572,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2024-05-30T14:57:08.162017979+08:00","start-ts":450114939267579905,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114939293794313} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":450114939700117540,"min-table-barrier-ts":450114939700117540,"admin-job-type":0} /tidb/cdc/default/default/task/position/7b7f6885-d525-4e15-b825-3fda4d9000b6/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683091896812572 {"id":7374683091896812572,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7374683091896812572 Namespace:default ID:ddl-manager SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:08.162017979 +0800 CST StartTs:450114939267579905 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004138bd0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114939293794313} {CheckpointTs:450114939687010317 MinTableBarrierTs:450114939687010317 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7b7f6885-d525-4e15-b825-3fda4d9000b6 {"id":"7b7f6885-d525-4e15-b825-3fda4d9000b6","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8497e63df 7b7f6885-d525-4e15-b825-3fda4d9000b6 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7374683091896812572,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2024-05-30T14:57:08.162017979+08:00","start-ts":450114939267579905,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114939293794313} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":450114939700117540,"min-table-barrier-ts":450114939700117540,"admin-job-type":0} /tidb/cdc/default/default/task/position/7b7f6885-d525-4e15-b825-3fda4d9000b6/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683091896812572 {"id":7374683091896812572,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7374683091896812572 Namespace:default ID:ddl-manager SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:08.162017979 +0800 CST StartTs:450114939267579905 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004138bd0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114939293794313} {CheckpointTs:450114939687010317 MinTableBarrierTs:450114939687010317 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7b7f6885-d525-4e15-b825-3fda4d9000b6 {"id":"7b7f6885-d525-4e15-b825-3fda4d9000b6","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8497e63df 7b7f6885-d525-4e15-b825-3fda4d9000b6 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7374683091896812572,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2024-05-30T14:57:08.162017979+08:00","start-ts":450114939267579905,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114939293794313} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":450114939700117540,"min-table-barrier-ts":450114939700117540,"admin-job-type":0} /tidb/cdc/default/default/task/position/7b7f6885-d525-4e15-b825-3fda4d9000b6/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683091896812572 {"id":7374683091896812572,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x table ddl_manager.finish_mark not exists for 1-th check, retry later wait process cdc.test exit for 3-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... wait process cdc.test exit for 4-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:22 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00429e1b0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114940612640777 MinTableBarrierTs:450114942971674630 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114940612640777, checkpointTs: 450114940612640777, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114940612640777, checkpointTs: 450114940612640777, state: Preparing span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114940612640777, checkpointTs: 450114940612640777, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/106332b6-86bf-4404-af9d-05eb62260dfb {"id":"106332b6-86bf-4404-af9d-05eb62260dfb","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8495259b5 106332b6-86bf-4404-af9d-05eb62260dfb /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114940612640777,"min-table-barrier-ts":450114942971674630,"admin-job-type":0} /tidb/cdc/default/default/task/position/106332b6-86bf-4404-af9d-05eb62260dfb/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00429e1b0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114940612640777 MinTableBarrierTs:450114942971674630 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114940612640777, checkpointTs: 450114940612640777, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114940612640777, checkpointTs: 450114940612640777, state: Preparing span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114940612640777, checkpointTs: 450114940612640777, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/106332b6-86bf-4404-af9d-05eb62260dfb {"id":"106332b6-86bf-4404-af9d-05eb62260dfb","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8495259b5 106332b6-86bf-4404-af9d-05eb62260dfb /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114940612640777,"min-table-barrier-ts":450114942971674630,"admin-job-type":0} /tidb/cdc/default/default/task/position/106332b6-86bf-4404-af9d-05eb62260dfb/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00429e1b0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114940612640777 MinTableBarrierTs:450114942971674630 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114940612640777, checkpointTs: 450114940612640777, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114940612640777, checkpointTs: 450114940612640777, state: Preparing span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114940612640777, checkpointTs: 450114940612640777, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/106332b6-86bf-4404-af9d-05eb62260dfb {"id":"106332b6-86bf-4404-af9d-05eb62260dfb","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8495259b5 106332b6-86bf-4404-af9d-05eb62260dfb /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114940612640777,"min-table-barrier-ts":450114942971674630,"admin-job-type":0} /tidb/cdc/default/default/task/position/106332b6-86bf-4404-af9d-05eb62260dfb/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 14:57:23 CST 2024] <<<<<< run test case changefeed_reconstruct success! >>>>>> cdc.test: no process found wait process cdc.test exit for 5-th time... process cdc.test already exit [Thu May 30 14:57:23 CST 2024] <<<<<< run test case gc_safepoint success! >>>>>> pass test: resume changefeed Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc) 3309751296 bytes in 5.37 secs (616211592 bytes/sec) [Pipeline] { table drop_tables.c not exists for 4-th check, retry later [Pipeline] cache ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 2-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2127ee4c0017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l00sh, pid:3990, start at 2024-05-30 14:57:22.862229355 +0800 CST m=+7.180317061 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:22.872 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:22.835 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:22.835 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2127ee4c0017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l00sh, pid:3990, start at 2024-05-30 14:57:22.862229355 +0800 CST m=+7.180317061 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:22.872 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:22.835 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:22.835 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2127daf80002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l00sh, pid:4058, start at 2024-05-30 14:57:21.601360682 +0800 CST m=+5.849638101 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:21.618 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:21.698 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:21.698 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cdc/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cdc/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/cdc/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cdc/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/cdc/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } "is-owner": true, run task successfully pass test: rebalance table Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release owner pid: 2981 owner id 106332b6-86bf-4404-af9d-05eb62260dfb [Thu May 30 14:57:25 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.30863088.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_hang_up_owner.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_hang_up_owner.server2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2127f7f0000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-gszd6, pid:1174, start at 2024-05-30 14:57:23.46429103 +0800 CST m=+6.051226912 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:23.473 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:23.452 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:23.452 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2127f7f0000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-gszd6, pid:1174, start at 2024-05-30 14:57:23.46429103 +0800 CST m=+6.051226912 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:23.473 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:23.452 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:23.452 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2127fb680013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-gszd6, pid:1236, start at 2024-05-30 14:57:23.72109084 +0800 CST m=+6.224888457 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:23.729 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:23.724 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:23.724 +0800 All versions after safe point can be accessed. (DO NOT EDIT) table ddl_manager.finish_mark not exists for 3-th check, retry later pass test: list processors Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/new_ci_collation/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/new_ci_collation/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/new_ci_collation/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/new_ci_collation/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/new_ci_collation/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table drop_tables.c not exists for 5-th check, retry later [Thu May 30 14:57:26 CST 2024] <<<<<< START cdc server in cdc case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc.53255327.out server --log-file /tmp/tidb_cdc_test/cdc/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cdc/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 4-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table drop_tables.c exists check diff successfully [Thu May 30 14:57:28 CST 2024] <<<<<< START cdc server in new_ci_collation case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.new_ci_collation.25582560.out server --log-file /tmp/tidb_cdc_test/new_ci_collation/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/new_ci_collation/cdc_data --cluster-id default + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:28 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003d023f0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114944544800778 MinTableBarrierTs:450114944544800778 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/106332b6-86bf-4404-af9d-05eb62260dfb {"id":"106332b6-86bf-4404-af9d-05eb62260dfb","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/cef3aae0-c060-46e6-a627-51289610c32a {"id":"cef3aae0-c060-46e6-a627-51289610c32a","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8495259b5 106332b6-86bf-4404-af9d-05eb62260dfb /tidb/cdc/default/__cdc_meta__/owner/22318fc849525a03 cef3aae0-c060-46e6-a627-51289610c32a /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114944544800778,"min-table-barrier-ts":450114944544800778,"admin-job-type":0} /tidb/cdc/default/default/task/position/106332b6-86bf-4404-af9d-05eb62260dfb/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/cef3aae0-c060-46e6-a627-51289610c32a/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003d023f0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114944544800778 MinTableBarrierTs:450114944544800778 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/106332b6-86bf-4404-af9d-05eb62260dfb {"id":"106332b6-86bf-4404-af9d-05eb62260dfb","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/cef3aae0-c060-46e6-a627-51289610c32a {"id":"cef3aae0-c060-46e6-a627-51289610c32a","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8495259b5 106332b6-86bf-4404-af9d-05eb62260dfb /tidb/cdc/default/__cdc_meta__/owner/22318fc849525a03 cef3aae0-c060-46e6-a627-51289610c32a /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114944544800778,"min-table-barrier-ts":450114944544800778,"admin-job-type":0} /tidb/cdc/default/default/task/position/106332b6-86bf-4404-af9d-05eb62260dfb/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/cef3aae0-c060-46e6-a627-51289610c32a/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003d023f0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114944544800778 MinTableBarrierTs:450114944544800778 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/106332b6-86bf-4404-af9d-05eb62260dfb {"id":"106332b6-86bf-4404-af9d-05eb62260dfb","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/cef3aae0-c060-46e6-a627-51289610c32a {"id":"cef3aae0-c060-46e6-a627-51289610c32a","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8495259b5 106332b6-86bf-4404-af9d-05eb62260dfb /tidb/cdc/default/__cdc_meta__/owner/22318fc849525a03 cef3aae0-c060-46e6-a627-51289610c32a /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114944544800778,"min-table-barrier-ts":450114944544800778,"admin-job-type":0} /tidb/cdc/default/default/task/position/106332b6-86bf-4404-af9d-05eb62260dfb/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/cef3aae0-c060-46e6-a627-51289610c32a/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list --server 'http://127.0.0.1:8301' 2>&1 | grep -v "106332b6-86bf-4404-af9d-05eb62260dfb" | grep id wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... table ddl_manager.finish_mark not exists for 5-th check, retry later wait process cdc.test exit for 3-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:29 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d2a6b4f4-fe73-4266-97b2-bbfcc637ed33 {"id":"d2a6b4f4-fe73-4266-97b2-bbfcc637ed33","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849c9bfc0 d2a6b4f4-fe73-4266-97b2-bbfcc637ed33 /tidb/cdc/default/default/upstream/7374683183093053460 {"id":7374683183093053460,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d2a6b4f4-fe73-4266-97b2-bbfcc637ed33 {"id":"d2a6b4f4-fe73-4266-97b2-bbfcc637ed33","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849c9bfc0 d2a6b4f4-fe73-4266-97b2-bbfcc637ed33 /tidb/cdc/default/default/upstream/7374683183093053460 {"id":7374683183093053460,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d2a6b4f4-fe73-4266-97b2-bbfcc637ed33 {"id":"d2a6b4f4-fe73-4266-97b2-bbfcc637ed33","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849c9bfc0 d2a6b4f4-fe73-4266-97b2-bbfcc637ed33 /tidb/cdc/default/default/upstream/7374683183093053460 {"id":7374683183093053460,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc.cli.5374.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ pass test: get processor "id": "cef3aae0-c060-46e6-a627-51289610c32a", "cluster-id": "default" "cluster-id": "default" run task successfully cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 14:57:30 CST 2024] <<<<<< run test case drop_many_tables success! >>>>>> pass test: move table table ddl_manager.finish_mark not exists for 6-th check, retry later capture_id: cef3aae0-c060-46e6-a627-51289610c32a ETCDCTL_API=3 etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix | grep 'cef3aae0-c060-46e6-a627-51289610c32a' cef3aae0-c060-46e6-a627-51289610c32a run task successfully test_hang_up_owner: pass ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Create changefeed successfully! ID: 8a740ec2-5ea2-4a85-97d5-e77cebc7aedb Info: {"upstream_id":7374683183093053460,"namespace":"default","id":"8a740ec2-5ea2-4a85-97d5-e77cebc7aedb","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T14:57:31.779765831+08:00","start_ts":450114945452605447,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114945452605447,"checkpoint_ts":450114945452605447,"checkpoint_time":"2024-05-30 14:57:31.635"} PASS + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:31 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d6f770ba-31e8-4451-ac14-29a59127a8a2 {"id":"d6f770ba-31e8-4451-ac14-29a59127a8a2","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849d074c1 d6f770ba-31e8-4451-ac14-29a59127a8a2 /tidb/cdc/default/default/upstream/7374683192025207747 {"id":7374683192025207747,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d6f770ba-31e8-4451-ac14-29a59127a8a2 {"id":"d6f770ba-31e8-4451-ac14-29a59127a8a2","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849d074c1 d6f770ba-31e8-4451-ac14-29a59127a8a2 /tidb/cdc/default/default/upstream/7374683192025207747 {"id":7374683192025207747,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d6f770ba-31e8-4451-ac14-29a59127a8a2 {"id":"d6f770ba-31e8-4451-ac14-29a59127a8a2","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849d074c1 d6f770ba-31e8-4451-ac14-29a59127a8a2 /tidb/cdc/default/default/upstream/7374683192025207747 {"id":7374683192025207747,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: f00619d2-ec4a-4c6f-aa81-28cd9a6835df Info: {"upstream_id":7374683192025207747,"namespace":"default","id":"f00619d2-ec4a-4c6f-aa81-28cd9a6835df","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?safe-mode=true","create_time":"2024-05-30T14:57:31.828013782+08:00","start_ts":450114944565772289,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114944565772289,"checkpoint_ts":450114944565772289,"checkpoint_time":"2024-05-30 14:57:28.252"} wait process cdc.test exit for 1-th time... coverage: 2.4% of statements in github.com/pingcap/tiflow/... pass test: set log level wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) pass test: remove changefeed cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit run test case test_expire_owner [Thu May 30 14:57:33 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.32283230.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_expire_owner.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_expire_owner.server1 --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading github.com/BurntSushi/toml v1.3.2 go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20240527072219-a97e6464c01d go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20240527072219-a97e6464c01d go: downloading github.com/pingcap/failpoint v0.0.0-20240412033321-fd0796e60f86 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading go.uber.org/atomic v1.11.0 go: downloading go.uber.org/multierr v1.11.0 table ddl_manager.finish_mark not exists for 7-th check, retry later pass test: resign owner =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/multi_capture/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/changefeed_pause_resume/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table ddl_manager.finish_mark not exists for 8-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading golang.org/x/text v0.15.0 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/pingcap/kvproto v0.0.0-20240522024016-df42997c2c57 go: downloading github.com/gorilla/mux v1.8.0 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/golang/protobuf v1.5.4 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20240510070134-e7f530c64790 go: downloading golang.org/x/sync v0.7.0 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/pingcap/tipb v0.0.0-20240507090649-2bf6bb0cb996 go: downloading github.com/prometheus/client_golang v1.19.1 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/danjacques/gofslock v0.0.0-20220131014315-6e321f4509c8 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/google/uuid v1.6.0 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading github.com/shirou/gopsutil/v3 v3.24.4 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/tikv/pd/client v0.0.0-20240210135946-3488a653ddd9 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading google.golang.org/grpc v1.64.0 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/aws/aws-sdk-go v1.44.259 go: downloading github.com/klauspost/compress v1.17.8 go: downloading cloud.google.com/go v0.112.2 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.6 go: downloading golang.org/x/oauth2 v0.18.0 go: downloading google.golang.org/api v0.170.0 go: downloading github.com/cockroachdb/pebble v0.0.0-20220415182917-06c9d3be25b3 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/stretchr/testify v1.9.0 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading golang.org/x/sys v0.20.0 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading github.com/google/btree v1.1.2 go: downloading golang.org/x/time v0.5.0 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/pkg/errors v0.9.1 go: downloading google.golang.org/protobuf v1.34.1 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec pass test: get tso start tidb cluster in /tmp/tidb_cdc_test/multi_capture Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading golang.org/x/net v0.25.0 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.1 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.1 start tidb cluster in /tmp/tidb_cdc_test/changefeed_pause_resume Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... go: downloading golang.org/x/crypto v0.23.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/prometheus/common v0.53.0 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:36 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004536000 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114946641690631 MinTableBarrierTs:450114946641690631 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114946641690631, checkpointTs: 450114946641690631, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114946641690631, checkpointTs: 450114946641690631, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114946641690631, checkpointTs: 450114946641690631, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0b1b31d0-c863-4be7-9e1f-d9eb28af12c8 {"id":"0b1b31d0-c863-4be7-9e1f-d9eb28af12c8","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525a4f 0b1b31d0-c863-4be7-9e1f-d9eb28af12c8 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114946641690631,"min-table-barrier-ts":450114946641690631,"admin-job-type":0} /tidb/cdc/default/default/task/position/0b1b31d0-c863-4be7-9e1f-d9eb28af12c8/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004536000 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114946641690631 MinTableBarrierTs:450114946641690631 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114946641690631, checkpointTs: 450114946641690631, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114946641690631, checkpointTs: 450114946641690631, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114946641690631, checkpointTs: 450114946641690631, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0b1b31d0-c863-4be7-9e1f-d9eb28af12c8 {"id":"0b1b31d0-c863-4be7-9e1f-d9eb28af12c8","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525a4f 0b1b31d0-c863-4be7-9e1f-d9eb28af12c8 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114946641690631,"min-table-barrier-ts":450114946641690631,"admin-job-type":0} /tidb/cdc/default/default/task/position/0b1b31d0-c863-4be7-9e1f-d9eb28af12c8/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004536000 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114946641690631 MinTableBarrierTs:450114946641690631 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114946641690631, checkpointTs: 450114946641690631, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114946641690631, checkpointTs: 450114946641690631, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114946641690631, checkpointTs: 450114946641690631, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0b1b31d0-c863-4be7-9e1f-d9eb28af12c8 {"id":"0b1b31d0-c863-4be7-9e1f-d9eb28af12c8","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525a4f 0b1b31d0-c863-4be7-9e1f-d9eb28af12c8 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114946641690631,"min-table-barrier-ts":450114946641690631,"admin-job-type":0} /tidb/cdc/default/default/task/position/0b1b31d0-c863-4be7-9e1f-d9eb28af12c8/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' go: downloading github.com/prometheus/procfs v0.15.0 go: downloading github.com/iancoleman/strcase v0.2.0 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/cockroachdb/redact v1.1.5 pass test: verify table go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/xitongsys/parquet-go v1.6.0 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/sasha-s/go-deadlock v0.3.1 go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading github.com/json-iterator/go v1.1.12 wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit # diff Configuration. check-thread-count = 4 export-fix-sql = true check-struct-only = false [task] output-dir = "/tmp/tidb_cdc_test/consistent_replicate_ddl/sync_diff/output" source-instances = ["tidb"] target-instance = "mysql" target-check-tables = ["consistent_replicate_ddl.usertable*","consistent_replicate_ddl.check*"] [data-sources] [data-sources.tidb] host = "127.0.0.1" port = 4000 user = "root" password = "" snapshot = "450114914392211463" [data-sources.mysql] host = "127.0.0.1" port = 3306 user = "root" password = "" [2024/05/30 14:57:35.425 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2024/05/30 14:57:35.426 +08:00] [INFO] [apply.go:78] ["set memory limit"] [memoryLimit=10737418240] [2024/05/30 14:57:35.426 +08:00] [INFO] [apply.go:94] ["Start http pprof server"] [addr=:6060] [2024/05/30 14:57:35.426 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2024/05/30 14:57:35.427 +08:00] [INFO] [redo.go:167] ["apply redo log starts"] [checkpointTs=450114913854816275] [resolvedTs=450114914392211463] [2024/05/30 14:57:35.428 +08:00] [INFO] [tz.go:36] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/05/30 14:57:35.428 +08:00] [WARN] [config.go:394] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/05/30 14:57:35.428 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=1.05488ms] [2024/05/30 14:57:35.428 +08:00] [ERROR] [config.go:209] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2024/05/30 14:57:35.428 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=1.598436ms] [2024/05/30 14:57:35.432 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22ONLY_FULL_GROUP_BY%2CNO_AUTO_VALUE_ON_ZERO%2CIGNORE_SPACE%2CNO_ENGINE_SUBSTITUTION%2CALLOW_INVALID_DATES%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/05/30 14:57:35.435 +08:00] [INFO] [mysql.go:198] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2024/05/30 14:57:35.436 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:57:35.436 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:57:35.436 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:57:35.436 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:57:35.436 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:57:35.436 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:57:35.436 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:57:35.436 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:57:35.436 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:57:35.436 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:57:35.436 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:57:35.436 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:57:35.436 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:57:35.436 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:57:35.436 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:57:35.436 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:57:35.436 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2024/05/30 14:57:35.436 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2024/05/30 14:57:35.436 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2024/05/30 14:57:35.436 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2024/05/30 14:57:35.436 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2024/05/30 14:57:35.436 +08:00] [INFO] [tz.go:36] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/05/30 14:57:35.436 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2024/05/30 14:57:35.436 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2024/05/30 14:57:35.436 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2024/05/30 14:57:35.436 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2024/05/30 14:57:35.436 +08:00] [WARN] [config.go:394] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/05/30 14:57:35.436 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2024/05/30 14:57:35.436 +08:00] [ERROR] [config.go:209] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2024/05/30 14:57:35.436 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2024/05/30 14:57:35.436 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2024/05/30 14:57:35.436 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2024/05/30 14:57:35.436 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2024/05/30 14:57:35.436 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2024/05/30 14:57:35.436 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2024/05/30 14:57:35.440 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22NO_ENGINE_SUBSTITUTION%2CALLOW_INVALID_DATES%2CONLY_FULL_GROUP_BY%2CNO_AUTO_VALUE_ON_ZERO%2CIGNORE_SPACE%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/05/30 14:57:35.442 +08:00] [INFO] [mysql_ddl_sink.go:99] ["MySQL DDL sink is created"] [namespace=] [changefeed=] [2024/05/30 14:57:35.442 +08:00] [WARN] [redo.go:288] ["apply DDL"] [ddl="{\"StartTs\":450114913854816270,\"CommitTs\":450114913854816275,\"Query\":\"DROP TABLE `consistent_replicate_ddl`.`usertable1`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable1\",\"tbl-id\":106,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":4,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [2024/05/30 14:57:35.442 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":450114913854816270,\"CommitTs\":450114913854816275,\"Query\":\"DROP TABLE `consistent_replicate_ddl`.`usertable1`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable1\",\"tbl-id\":106,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":4,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [namespace=] [changefeed=] [2024/05/30 14:57:35.649 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="DROP TABLE `consistent_replicate_ddl`.`usertable1`"] [duration=206.384655ms] [namespace=] [changefeed=] [2024/05/30 14:57:35.649 +08:00] [WARN] [redo.go:288] ["apply DDL"] [ddl="{\"StartTs\":450114913907245059,\"CommitTs\":450114913907245062,\"Query\":\"RENAME TABLE consistent_replicate_ddl.usertable_bak TO consistent_replicate_ddl.usertable1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable1\",\"tbl-id\":112,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":14,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [2024/05/30 14:57:35.649 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":450114913907245059,\"CommitTs\":450114913907245062,\"Query\":\"RENAME TABLE consistent_replicate_ddl.usertable_bak TO consistent_replicate_ddl.usertable1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable1\",\"tbl-id\":112,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":14,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [namespace=] [changefeed=] [2024/05/30 14:57:35.790 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="RENAME TABLE consistent_replicate_ddl.usertable_bak TO consistent_replicate_ddl.usertable1"] [duration=140.915848ms] [namespace=] [changefeed=] [2024/05/30 14:57:35.790 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=282] [newQuota=72192] [2024/05/30 14:57:35.790 +08:00] [WARN] [redo.go:288] ["apply DDL"] [ddl="{\"StartTs\":450114913946566666,\"CommitTs\":450114913946566669,\"Query\":\"RENAME TABLE consistent_replicate_ddl.usertable1 TO consistent_replicate_ddl.usertable1_1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable1_1\",\"tbl-id\":112,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":14,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [2024/05/30 14:57:35.991 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":450114913946566666,\"CommitTs\":450114913946566669,\"Query\":\"RENAME TABLE consistent_replicate_ddl.usertable1 TO consistent_replicate_ddl.usertable1_1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable1_1\",\"tbl-id\":112,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":14,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [namespace=] [changefeed=] [2024/05/30 14:57:36.127 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="RENAME TABLE consistent_replicate_ddl.usertable1 TO consistent_replicate_ddl.usertable1_1"] [duration=136.151441ms] [namespace=] [changefeed=] [2024/05/30 14:57:36.127 +08:00] [WARN] [redo.go:288] ["apply DDL"] [ddl="{\"StartTs\":450114913985888279,\"CommitTs\":450114913985888283,\"Query\":\"CREATE DATABASE consistent_replicate_ddl1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl1\",\"tbl-name\":\"\",\"tbl-id\":0,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":1,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [2024/05/30 14:57:36.328 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":450114913985888279,\"CommitTs\":450114913985888283,\"Query\":\"CREATE DATABASE consistent_replicate_ddl1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl1\",\"tbl-name\":\"\",\"tbl-id\":0,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":1,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [namespace=] [changefeed=] [2024/05/30 14:57:36.403 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="CREATE DATABASE consistent_replicate_ddl1"] [duration=75.259502ms] [namespace=] [changefeed=] [2024/05/30 14:57:36.403 +08:00] [WARN] [redo.go:288] ["apply DDL"] [ddl="{\"StartTs\":450114914012102684,\"CommitTs\":450114914012102688,\"Query\":\"ALTER DATABASE consistent_replicate_ddl CHARACTER SET utf8mb4 COLLATE utf8mb4_bin\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"\",\"tbl-id\":0,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":26,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [2024/05/30 14:57:36.403 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":450114914012102684,\"CommitTs\":450114914012102688,\"Query\":\"ALTER DATABASE consistent_replicate_ddl CHARACTER SET utf8mb4 COLLATE utf8mb4_bin\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"\",\"tbl-id\":0,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":26,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [namespace=] [changefeed=] [2024/05/30 14:57:36.475 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="ALTER DATABASE consistent_replicate_ddl CHARACTER SET utf8mb4 COLLATE utf8mb4_bin"] [duration=71.462637ms] [namespace=] [changefeed=] [2024/05/30 14:57:36.475 +08:00] [WARN] [redo.go:288] ["apply DDL"] [ddl="{\"StartTs\":450114914038317075,\"CommitTs\":450114914038317080,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable2 CHARACTER SET utf8mb4 COLLATE utf8mb4_bin\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable2\",\"tbl-id\":108,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":22,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [2024/05/30 14:57:36.475 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":450114914038317075,\"CommitTs\":450114914038317080,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable2 CHARACTER SET utf8mb4 COLLATE utf8mb4_bin\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable2\",\"tbl-id\":108,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":22,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [namespace=] [changefeed=] [2024/05/30 14:57:36.547 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="ALTER TABLE consistent_replicate_ddl.usertable2 CHARACTER SET utf8mb4 COLLATE utf8mb4_bin"] [duration=72.493209ms] [namespace=] [changefeed=] [2024/05/30 14:57:36.547 +08:00] [WARN] [redo.go:288] ["apply DDL"] [ddl="{\"StartTs\":450114914117222410,\"CommitTs\":450114914130067458,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable2 DROP COLUMN FIELD0\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable2\",\"tbl-id\":108,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":6,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [2024/05/30 14:57:36.749 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":450114914117222410,\"CommitTs\":450114914130067458,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable2 DROP COLUMN FIELD0\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable2\",\"tbl-id\":108,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":6,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [namespace=] [changefeed=] [2024/05/30 14:57:37.000 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="ALTER TABLE consistent_replicate_ddl.usertable2 DROP COLUMN FIELD0"] [duration=251.665474ms] [namespace=] [changefeed=] [2024/05/30 14:57:37.000 +08:00] [WARN] [redo.go:288] ["apply DDL"] [ddl="{\"StartTs\":450114914195603466,\"CommitTs\":450114914195603471,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable2 ADD COLUMN dummy varchar(30)\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable2\",\"tbl-id\":108,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":5,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [2024/05/30 14:57:37.000 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":450114914195603466,\"CommitTs\":450114914195603471,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable2 ADD COLUMN dummy varchar(30)\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable2\",\"tbl-id\":108,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":5,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [namespace=] [changefeed=] [2024/05/30 14:57:37.256 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="ALTER TABLE consistent_replicate_ddl.usertable2 ADD COLUMN dummy varchar(30)"] [duration=255.184409ms] [namespace=] [changefeed=] [2024/05/30 14:57:37.256 +08:00] [WARN] [redo.go:288] ["apply DDL"] [ddl="{\"StartTs\":450114914261401607,\"CommitTs\":450114914261401611,\"Query\":\"DROP DATABASE consistent_replicate_ddl1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl1\",\"tbl-name\":\"\",\"tbl-id\":0,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":2,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [2024/05/30 14:57:37.256 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":450114914261401607,\"CommitTs\":450114914261401611,\"Query\":\"DROP DATABASE consistent_replicate_ddl1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl1\",\"tbl-name\":\"\",\"tbl-id\":0,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":2,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [namespace=] [changefeed=] [2024/05/30 14:57:37.447 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="DROP DATABASE consistent_replicate_ddl1"] [duration=191.111978ms] [namespace=] [changefeed=] [2024/05/30 14:57:37.447 +08:00] [WARN] [redo.go:288] ["apply DDL"] [ddl="{\"StartTs\":450114914287353875,\"CommitTs\":450114914287353879,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable3 CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable3\",\"tbl-id\":110,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":22,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [2024/05/30 14:57:37.447 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":450114914287353875,\"CommitTs\":450114914287353879,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable3 CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable3\",\"tbl-id\":110,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":22,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [namespace=] [changefeed=] [2024/05/30 14:57:37.521 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="ALTER TABLE consistent_replicate_ddl.usertable3 CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci"] [duration=73.710242ms] [namespace=] [changefeed=] [2024/05/30 14:57:37.521 +08:00] [WARN] [redo.go:288] ["apply DDL"] [ddl="{\"StartTs\":450114914326675470,\"CommitTs\":450114914326675474,\"Query\":\"RENAME TABLE `consistent_replicate_ddl`.`usertable3` TO `consistent_replicate_ddl`.`usertable3_1`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable3_1\",\"tbl-id\":110,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":14,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [2024/05/30 14:57:37.521 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":450114914326675470,\"CommitTs\":450114914326675474,\"Query\":\"RENAME TABLE `consistent_replicate_ddl`.`usertable3` TO `consistent_replicate_ddl`.`usertable3_1`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable3_1\",\"tbl-id\":110,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":14,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [namespace=] [changefeed=] [2024/05/30 14:57:37.656 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="RENAME TABLE `consistent_replicate_ddl`.`usertable3` TO `consistent_replicate_ddl`.`usertable3_1`"] [duration=135.31042ms] [namespace=] [changefeed=] [2024/05/30 14:57:37.656 +08:00] [WARN] [redo.go:288] ["apply DDL"] [ddl="{\"StartTs\":450114914326675470,\"CommitTs\":450114914326675474,\"Query\":\"RENAME TABLE `consistent_replicate_ddl`.`usertable2` TO `consistent_replicate_ddl`.`usertable2_1`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable2_1\",\"tbl-id\":108,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":14,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [2024/05/30 14:57:37.656 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":450114914326675470,\"CommitTs\":450114914326675474,\"Query\":\"RENAME TABLE `consistent_replicate_ddl`.`usertable2` TO `consistent_replicate_ddl`.`usertable2_1`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable2_1\",\"tbl-id\":108,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":14,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [namespace=] [changefeed=] go: downloading github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/kr/pretty v0.3.1 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240521202816-d264139d666e go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/pingcap/badger v1.5.1-0.20230103063557-828f39b09b6d go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/fatih/color v1.16.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/mattn/go-runewidth v0.0.15 go: downloading github.com/apache/thrift v0.16.0 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/petermattis/goid v0.0.0-20231207134359-e60b3f734c67 go: downloading github.com/golang-jwt/jwt/v5 v5.2.0 table ddl_manager.finish_mark not exists for 9-th check, retry later [2024/05/30 14:57:37.806 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="RENAME TABLE `consistent_replicate_ddl`.`usertable2` TO `consistent_replicate_ddl`.`usertable2_1`"] [duration=150.272172ms] [namespace=] [changefeed=] [2024/05/30 14:57:37.807 +08:00] [WARN] [redo.go:288] ["apply DDL"] [ddl="{\"StartTs\":450114914352889879,\"CommitTs\":450114914352889882,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable3_1 MODIFY COLUMN FIELD1 varchar(100)\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable3_1\",\"tbl-id\":110,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":12,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [2024/05/30 14:57:37.807 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":450114914352889879,\"CommitTs\":450114914352889882,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable3_1 MODIFY COLUMN FIELD1 varchar(100)\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable3_1\",\"tbl-id\":110,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":12,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [namespace=] [changefeed=] [2024/05/30 14:57:37.881 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="ALTER TABLE consistent_replicate_ddl.usertable3_1 MODIFY COLUMN FIELD1 varchar(100)"] [duration=74.890865ms] [namespace=] [changefeed=] [2024/05/30 14:57:37.882 +08:00] [WARN] [redo.go:288] ["apply DDL"] [ddl="{\"StartTs\":450114914392211459,\"CommitTs\":450114914392211463,\"Query\":\"CREATE table consistent_replicate_ddl.check1(id int primary key)\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"check1\",\"tbl-id\":127,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":3,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2128bec00012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t5729, pid:3756, start at 2024-05-30 14:57:36.202403508 +0800 CST m=+8.774643333 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:36.210 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:36.176 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:36.176 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2128bec00012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t5729, pid:3756, start at 2024-05-30 14:57:36.202403508 +0800 CST m=+8.774643333 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:36.210 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:36.176 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:36.176 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2128bf90000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t5729, pid:3839, start at 2024-05-30 14:57:36.244400575 +0800 CST m=+8.717625566 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:36.253 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:36.228 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:36.228 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_error/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_error/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.12.0 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/DataDog/zstd v1.5.5 go: downloading github.com/lestrrat-go/httpcc v1.0.1 [2024/05/30 14:57:38.083 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":450114914392211459,\"CommitTs\":450114914392211463,\"Query\":\"CREATE table consistent_replicate_ddl.check1(id int primary key)\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"check1\",\"tbl-id\":127,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":3,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [namespace=] [changefeed=] [2024/05/30 14:57:38.161 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="CREATE table consistent_replicate_ddl.check1(id int primary key)"] [duration=78.454634ms] [namespace=] [changefeed=] [2024/05/30 14:57:38.161 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}] [checkpointTs=450114914392211463] [2024/05/30 14:57:38.161 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}] [checkpointTs=450114914392211463] go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading github.com/go-logr/logr v1.4.1 go: downloading github.com/go-logr/stdr v1.2.2 "is-owner": true, run task successfully [2024/05/30 14:57:38.262 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}] [checkpointTs=450114914392211463] [2024/05/30 14:57:38.262 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}] [checkpointTs=450114914392211463] [2024/05/30 14:57:38.362 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}] [checkpointTs=450114914392211463] [2024/05/30 14:57:38.362 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}] [checkpointTs=450114914392211463] table new_ci_collation_test.t1 exists table new_ci_collation_test.t2 exists table new_ci_collation_test.t3 exists table new_ci_collation_test.t4 not exists for 1-th check, retry later [2024/05/30 14:57:38.462 +08:00] [INFO] [redo.go:223] ["apply redo log finishes"] [appliedLogCount=101] [appliedDDLCount=14] [currentCheckpoint=450114914392211463] [2024/05/30 14:57:38.462 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2024/05/30 14:57:38.462 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2024/05/30 14:57:38.462 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2024/05/30 14:57:38.462 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2024/05/30 14:57:38.462 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2024/05/30 14:57:38.462 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] [2024/05/30 14:57:38.462 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2024/05/30 14:57:38.462 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2024/05/30 14:57:38.462 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2024/05/30 14:57:38.462 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2024/05/30 14:57:38.462 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2024/05/30 14:57:38.462 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2024/05/30 14:57:38.462 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] [2024/05/30 14:57:38.462 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2024/05/30 14:57:38.462 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2024/05/30 14:57:38.462 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] Apply redo log successfully table consistent_replicate_ddl.check1 exists check diff successfully [Thu May 30 14:57:38 CST 2024] <<<<<< run test case consistent_replicate_ddl success! >>>>>> count(*) 50 pass test: create changefeed v2 table ddl_manager.finish_mark not exists for 10-th check, retry later owner pid: 3233 owner id 0b1b31d0-c863-4be7-9e1f-d9eb28af12c8 process status: T ETCDCTL_API=3 etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix | grep -v '0b1b31d0-c863-4be7-9e1f-d9eb28af12c8' /tidb/cdc/default/__cdc_meta__/owner/22318fc849525a4f run task successfully process status: S cdc.test cli capture list 2>&1 | grep '"is-owner": true' + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5214.out cli tso query --pd=http://127.0.0.1:2379 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/processor_stop_delay/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table new_ci_collation_test.t4 exists table new_ci_collation_test.t5 not exists for 1-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release pass test: remove changefeed v2 ***************** properties ***************** "readproportion"="0" "scanproportion"="0" "threadcount"="2" "updateproportion"="0" "mysql.db"="testSync" "recordcount"="10" "readallfields"="true" "operationcount"="0" "workload"="core" "mysql.port"="4000" "mysql.user"="root" "requestdistribution"="uniform" "mysql.host"="127.0.0.1" "insertproportion"="0" "dotransactions"="false" ********************************************** Run finished, takes 4.742132ms INSERT - Takes(s): 0.0, Count: 10, OPS: 3287.0, Avg(us): 868, Min(us): 526, Max(us): 1846, 95th(us): 2000, 99th(us): 2000 table ddl_manager.finish_mark not exists for 11-th check, retry later + set +x + tso='450114947678470148 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114947678470148 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ***************** properties ***************** "recordcount"="20" "mysql.db"="changefeed_error" "mysql.host"="127.0.0.1" "threadcount"="4" "updateproportion"="0" "operationcount"="0" "mysql.user"="root" "scanproportion"="0" "readallfields"="true" "insertproportion"="0" "mysql.port"="4000" "readproportion"="0" "dotransactions"="false" "workload"="core" "requestdistribution"="uniform" ********************************************** Run finished, takes 10.501112ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3473.5, Avg(us): 1991, Min(us): 967, Max(us): 4674, 95th(us): 5000, 99th(us): 5000 [Thu May 30 14:57:41 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/NewChangefeedNoRetryError=1*return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.52675269.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 "is-owner": true, run task successfully test_expire_owner pass table new_ci_collation_test.t5 exists check diff failed 1-th time, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading github.com/jmespath/go-jmespath v0.4.0 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading github.com/google/s2a-go v0.1.7 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading github.com/felixge/httpsnoop v1.0.4 status code 200 pass test: show metadata status code 200 status code 200 pass test: delete service_gc_safepoint status code 500 pass test: resolve lock Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... start tidb cluster in /tmp/tidb_cdc_test/processor_stop_delay Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_owner_cleanup_stale_tasks [Thu May 30 14:57:43 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.33933395.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_owner_cleanup_stale_tasks.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_owner_cleanup_stale_tasks.server1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 1-th time... table ddl_manager.finish_mark not exists for 12-th check, retry later wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 3-th time... check diff failed 2-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:44 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fe1cd9c3-1823-4c69-83ea-1db1e01e0218 {"id":"fe1cd9c3-1823-4c69-83ea-1db1e01e0218","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849f527e0 fe1cd9c3-1823-4c69-83ea-1db1e01e0218 /tidb/cdc/default/default/upstream/7374683233227573581 {"id":7374683233227573581,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fe1cd9c3-1823-4c69-83ea-1db1e01e0218 {"id":"fe1cd9c3-1823-4c69-83ea-1db1e01e0218","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849f527e0 fe1cd9c3-1823-4c69-83ea-1db1e01e0218 /tidb/cdc/default/default/upstream/7374683233227573581 {"id":7374683233227573581,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fe1cd9c3-1823-4c69-83ea-1db1e01e0218 {"id":"fe1cd9c3-1823-4c69-83ea-1db1e01e0218","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849f527e0 fe1cd9c3-1823-4c69-83ea-1db1e01e0218 /tidb/cdc/default/default/upstream/7374683233227573581 {"id":7374683233227573581,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5328.out cli changefeed create --start-ts=450114947678470148 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' -c changefeed-error cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 14:57:45 CST 2024] <<<<<< run test case http_api_tls success! >>>>>> Create changefeed successfully! ID: changefeed-error Info: {"upstream_id":7374683233227573581,"namespace":"default","id":"changefeed-error","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T14:57:45.386750116+08:00","start_ts":450114947678470148,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114947678470148,"checkpoint_ts":450114947678470148,"checkpoint_time":"2024-05-30 14:57:40.126"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 13-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 3-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-error failed [CDC:ErrStartTsBeforeGC] + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error + expected_state=failed + error_msg='[CDC:ErrStartTsBeforeGC]' + tls_dir='[CDC:ErrStartTsBeforeGC]' + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error -s Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:46 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037533b0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114947965517825 MinTableBarrierTs:450114949276237827 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114947965517825, checkpointTs: 450114947965517825, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114947965517825, checkpointTs: 450114947965517825, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114947965517825, checkpointTs: 450114947965517825, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3efaf4bd-8e70-4182-bd87-02fec40a23fc {"id":"3efaf4bd-8e70-4182-bd87-02fec40a23fc","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525aaf 3efaf4bd-8e70-4182-bd87-02fec40a23fc /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114947965517825,"min-table-barrier-ts":450114949276237827,"admin-job-type":0} /tidb/cdc/default/default/task/position/3efaf4bd-8e70-4182-bd87-02fec40a23fc/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037533b0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114947965517825 MinTableBarrierTs:450114949276237827 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114947965517825, checkpointTs: 450114947965517825, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114947965517825, checkpointTs: 450114947965517825, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114947965517825, checkpointTs: 450114947965517825, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3efaf4bd-8e70-4182-bd87-02fec40a23fc {"id":"3efaf4bd-8e70-4182-bd87-02fec40a23fc","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525aaf 3efaf4bd-8e70-4182-bd87-02fec40a23fc /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114947965517825,"min-table-barrier-ts":450114949276237827,"admin-job-type":0} /tidb/cdc/default/default/task/position/3efaf4bd-8e70-4182-bd87-02fec40a23fc/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037533b0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114947965517825 MinTableBarrierTs:450114949276237827 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114947965517825, checkpointTs: 450114947965517825, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114947965517825, checkpointTs: 450114947965517825, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114947965517825, checkpointTs: 450114947965517825, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3efaf4bd-8e70-4182-bd87-02fec40a23fc {"id":"3efaf4bd-8e70-4182-bd87-02fec40a23fc","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525aaf 3efaf4bd-8e70-4182-bd87-02fec40a23fc /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114947965517825,"min-table-barrier-ts":450114949276237827,"admin-job-type":0} /tidb/cdc/default/default/task/position/3efaf4bd-8e70-4182-bd87-02fec40a23fc/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' + info='{ "upstream_id": 7374683233227573581, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 450114947678470148, "checkpoint_time": "2024-05-30 14:57:40.126", "error": { "time": "2024-05-30T14:57:45.495922907+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 450114947678469848 is earlier than or equal to GC safepoint at 450114947678470148" } }' + echo '{ "upstream_id": 7374683233227573581, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 450114947678470148, "checkpoint_time": "2024-05-30 14:57:40.126", "error": { "time": "2024-05-30T14:57:45.495922907+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 450114947678469848 is earlier than or equal to GC safepoint at 450114947678470148" } }' { "upstream_id": 7374683233227573581, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 450114947678470148, "checkpoint_time": "2024-05-30 14:57:40.126", "error": { "time": "2024-05-30T14:57:45.495922907+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 450114947678469848 is earlier than or equal to GC safepoint at 450114947678470148" } } ++ jq -r .state ++ echo '{' '"upstream_id":' 7374683233227573581, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"failed",' '"checkpoint_tso":' 450114947678470148, '"checkpoint_time":' '"2024-05-30' '14:57:40.126",' '"error":' '{' '"time":' '"2024-05-30T14:57:45.495922907+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]fail' to create or maintain changefeed because start-ts 450114947678469848 is earlier than or equal to GC safepoint at '450114947678470148"' '}' '}' + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ echo '{' '"upstream_id":' 7374683233227573581, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"failed",' '"checkpoint_tso":' 450114947678470148, '"checkpoint_time":' '"2024-05-30' '14:57:40.126",' '"error":' '{' '"time":' '"2024-05-30T14:57:45.495922907+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]fail' to create or maintain changefeed because start-ts 450114947678469848 is earlier than or equal to GC safepoint at '450114947678470148"' '}' '}' ++ jq -r .error.message + message='[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 450114947678469848 is earlier than or equal to GC safepoint at 450114947678470148' + [[ ! [CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 450114947678469848 is earlier than or equal to GC safepoint at 450114947678470148 =~ \[CDC:ErrStartTsBeforeGC] ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5390.out cli changefeed resume -c changefeed-error PASS table ddl_manager.finish_mark not exists for 14-th check, retry later coverage: 2.0% of statements in github.com/pingcap/tiflow/... "is-owner": true, run task successfully check diff failed 4-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21296f080018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:4365, start at 2024-05-30 14:57:47.506272228 +0800 CST m=+5.254700750 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:47.514 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:47.508 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:47.508 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21296e580018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-htrwd, pid:6892, start at 2024-05-30 14:57:47.452587987 +0800 CST m=+5.192879361 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:47.460 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:47.464 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:47.464 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21296e580018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-htrwd, pid:6892, start at 2024-05-30 14:57:47.452587987 +0800 CST m=+5.192879361 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:47.460 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:47.464 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:47.464 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212970a80014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-htrwd, pid:6968, start at 2024-05-30 14:57:47.588849526 +0800 CST m=+5.246824092 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:47.596 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:47.562 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:47.562 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_capture/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_capture/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x table changefeed_error.usertable not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 15-th check, retry later owner pid: 3398 owner id 3efaf4bd-8e70-4182-bd87-02fec40a23fc [Thu May 30 14:57:49 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.35003502.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_owner_cleanup_stale_tasks.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_owner_cleanup_stale_tasks.server2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21296f080018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:4365, start at 2024-05-30 14:57:47.506272228 +0800 CST m=+5.254700750 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:47.514 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:47.508 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:47.508 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21296ee80015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:4450, start at 2024-05-30 14:57:47.497094811 +0800 CST m=+5.166376812 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:47.504 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:47.500 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:47.500 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/consistent_replicate_gbk/run.sh using Sink-Type: mysql... <<================= * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:24927; Connection refused * Closing connection 0 You are running an older version of MinIO released 3 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.cli.8227.out cli tso query --pd=http://127.0.0.1:2379 TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 5-th time, retry later table changefeed_error.usertable exists check diff failed 1-th time, retry later * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:24927 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 17D431F180C5A8EF < X-Xss-Protection: 1; mode=block < Date: Thu, 30 May 2024 06:57:51 GMT < { [data not shown] * Connection #0 to host 127.0.0.1 left intact Bucket 's3://logbucket/' created The 1 times to try to start tidb cluster... table ddl_manager.finish_mark not exists for 16-th check, retry later + set +x + tso='450114950519586817 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114950519586817 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ***************** properties ***************** "operationcount"="0" "threadcount"="2" "mysql.host"="127.0.0.1" "mysql.db"="multi_capture_1" "requestdistribution"="uniform" "insertproportion"="0" "workload"="core" "scanproportion"="0" "mysql.port"="4000" "dotransactions"="false" "mysql.user"="root" "readproportion"="0" "readallfields"="true" "updateproportion"="0" "recordcount"="10" ********************************************** Run finished, takes 8.477407ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2212.1, Avg(us): 1614, Min(us): 897, Max(us): 3857, 95th(us): 4000, 99th(us): 4000 [Thu May 30 14:57:52 CST 2024] <<<<<< START cdc server in changefeed_pause_resume case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_pause_resume.56975699.out server --log-file /tmp/tidb_cdc_test/changefeed_pause_resume/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_pause_resume/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ***************** properties ***************** "recordcount"="10" "scanproportion"="0" "mysql.port"="4000" "requestdistribution"="uniform" "readallfields"="true" "readproportion"="0" "mysql.host"="127.0.0.1" "mysql.user"="root" "dotransactions"="false" "operationcount"="0" "mysql.db"="multi_capture_2" "workload"="core" "insertproportion"="0" "threadcount"="2" "updateproportion"="0" ********************************************** Run finished, takes 9.097014ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2065.7, Avg(us): 1726, Min(us): 999, Max(us): 4153, 95th(us): 5000, 99th(us): 5000 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:52 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0033c9560 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114950849363974 MinTableBarrierTs:450114950849363974 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3efaf4bd-8e70-4182-bd87-02fec40a23fc {"id":"3efaf4bd-8e70-4182-bd87-02fec40a23fc","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/deb04e05-92c9-47c8-b5ce-55866dd224cf {"id":"deb04e05-92c9-47c8-b5ce-55866dd224cf","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525aaf 3efaf4bd-8e70-4182-bd87-02fec40a23fc /tidb/cdc/default/__cdc_meta__/owner/22318fc849525af1 deb04e05-92c9-47c8-b5ce-55866dd224cf /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114950849363974,"min-table-barrier-ts":450114950849363974,"admin-job-type":0} /tidb/cdc/default/default/task/position/3efaf4bd-8e70-4182-bd87-02fec40a23fc/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/deb04e05-92c9-47c8-b5ce-55866dd224cf/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0033c9560 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114950849363974 MinTableBarrierTs:450114950849363974 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3efaf4bd-8e70-4182-bd87-02fec40a23fc {"id":"3efaf4bd-8e70-4182-bd87-02fec40a23fc","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/deb04e05-92c9-47c8-b5ce-55866dd224cf {"id":"deb04e05-92c9-47c8-b5ce-55866dd224cf","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525aaf 3efaf4bd-8e70-4182-bd87-02fec40a23fc /tidb/cdc/default/__cdc_meta__/owner/22318fc849525af1 deb04e05-92c9-47c8-b5ce-55866dd224cf /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114950849363974,"min-table-barrier-ts":450114950849363974,"admin-job-type":0} /tidb/cdc/default/default/task/position/3efaf4bd-8e70-4182-bd87-02fec40a23fc/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/deb04e05-92c9-47c8-b5ce-55866dd224cf/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0033c9560 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114950849363974 MinTableBarrierTs:450114950849363974 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3efaf4bd-8e70-4182-bd87-02fec40a23fc {"id":"3efaf4bd-8e70-4182-bd87-02fec40a23fc","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/deb04e05-92c9-47c8-b5ce-55866dd224cf {"id":"deb04e05-92c9-47c8-b5ce-55866dd224cf","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525aaf 3efaf4bd-8e70-4182-bd87-02fec40a23fc /tidb/cdc/default/__cdc_meta__/owner/22318fc849525af1 deb04e05-92c9-47c8-b5ce-55866dd224cf /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114950849363974,"min-table-barrier-ts":450114950849363974,"admin-job-type":0} /tidb/cdc/default/default/task/position/3efaf4bd-8e70-4182-bd87-02fec40a23fc/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/deb04e05-92c9-47c8-b5ce-55866dd224cf/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list --server 'http://127.0.0.1:8301' 2>&1 | grep -v "3efaf4bd-8e70-4182-bd87-02fec40a23fc" | grep id ***************** properties ***************** "readproportion"="0" "insertproportion"="0" "recordcount"="10" "mysql.db"="multi_capture_3" "threadcount"="2" "updateproportion"="0" "mysql.host"="127.0.0.1" "mysql.port"="4000" "workload"="core" "readallfields"="true" "scanproportion"="0" "operationcount"="0" "mysql.user"="root" "requestdistribution"="uniform" "dotransactions"="false" ********************************************** Run finished, takes 9.020678ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2075.2, Avg(us): 1717, Min(us): 940, Max(us): 4106, 95th(us): 5000, 99th(us): 5000 ***************** properties ***************** "threadcount"="2" "requestdistribution"="uniform" "readproportion"="0" "updateproportion"="0" "mysql.user"="root" "mysql.db"="multi_capture_4" "readallfields"="true" "scanproportion"="0" "workload"="core" "insertproportion"="0" "mysql.port"="4000" "dotransactions"="false" "mysql.host"="127.0.0.1" "operationcount"="0" "recordcount"="10" ********************************************** Run finished, takes 9.189851ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2107.8, Avg(us): 1762, Min(us): 931, Max(us): 4340, 95th(us): 5000, 99th(us): 5000 [Thu May 30 14:57:53 CST 2024] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.83448346.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff successfully ***************** properties ***************** "mysql.db"="changefeed_error" "dotransactions"="false" "readallfields"="true" "threadcount"="4" "requestdistribution"="uniform" "mysql.port"="4000" "recordcount"="20" "readproportion"="0" "scanproportion"="0" "updateproportion"="0" "mysql.user"="root" "workload"="core" "insertproportion"="0" "mysql.host"="127.0.0.1" "operationcount"="0" ********************************************** Run finished, takes 5.762907ms INSERT - Takes(s): 0.0, Count: 20, OPS: 4935.7, Avg(us): 952, Min(us): 528, Max(us): 2376, 95th(us): 3000, 99th(us): 3000 check diff successfully {"id":"fe1cd9c3-1823-4c69-83ea-1db1e01e0218","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/fe1cd9c3-1823-4c69-83ea-1db1e01e0218 + [[ /tidb/cdc/default/__cdc_meta__/capture/fe1cd9c3-1823-4c69-83ea-1db1e01e0218 =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/fe1cd9c3-1823-4c69-83ea-1db1e01e0218' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/fe1cd9c3-1823-4c69-83ea-1db1e01e0218 + echo 'check failed' check failed + exit 1 run task failed 1-th time, retry later table ddl_manager.finish_mark not exists for 17-th check, retry later "cluster-id": "default" "id": "deb04e05-92c9-47c8-b5ce-55866dd224cf", "cluster-id": "default" run task successfully start tidb cluster in /tmp/tidb_cdc_test/consistent_replicate_gbk Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... check diff successfully check diff failed 1-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/api_v2/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/fe1cd9c3-1823-4c69-83ea-1db1e01e0218 + [[ /tidb/cdc/default/__cdc_meta__/capture/fe1cd9c3-1823-4c69-83ea-1db1e01e0218 =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/fe1cd9c3-1823-4c69-83ea-1db1e01e0218' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/fe1cd9c3-1823-4c69-83ea-1db1e01e0218 + echo 'check failed' check failed + exit 1 run task failed 2-th time, retry later table ddl_manager.finish_mark not exists for 18-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:55 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/927f37db-3fe0-458b-ba57-42d10eacca78 {"id":"927f37db-3fe0-458b-ba57-42d10eacca78","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a318cbd 927f37db-3fe0-458b-ba57-42d10eacca78 /tidb/cdc/default/default/upstream/7374683297938007148 {"id":7374683297938007148,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/927f37db-3fe0-458b-ba57-42d10eacca78 {"id":"927f37db-3fe0-458b-ba57-42d10eacca78","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a318cbd 927f37db-3fe0-458b-ba57-42d10eacca78 /tidb/cdc/default/default/upstream/7374683297938007148 {"id":7374683297938007148,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/927f37db-3fe0-458b-ba57-42d10eacca78 {"id":"927f37db-3fe0-458b-ba57-42d10eacca78","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a318cbd 927f37db-3fe0-458b-ba57-42d10eacca78 /tidb/cdc/default/default/upstream/7374683297938007148 {"id":7374683297938007148,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:56 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e95abbc6-5001-4f8c-896a-f50ad7085d4b {"id":"e95abbc6-5001-4f8c-896a-f50ad7085d4b","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a319525 e95abbc6-5001-4f8c-896a-f50ad7085d4b /tidb/cdc/default/default/upstream/7374683301044003426 {"id":7374683301044003426,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e95abbc6-5001-4f8c-896a-f50ad7085d4b {"id":"e95abbc6-5001-4f8c-896a-f50ad7085d4b","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a319525 e95abbc6-5001-4f8c-896a-f50ad7085d4b /tidb/cdc/default/default/upstream/7374683301044003426 {"id":7374683301044003426,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e95abbc6-5001-4f8c-896a-f50ad7085d4b {"id":"e95abbc6-5001-4f8c-896a-f50ad7085d4b","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a319525 e95abbc6-5001-4f8c-896a-f50ad7085d4b /tidb/cdc/default/default/upstream/7374683301044003426 {"id":7374683301044003426,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Thu May 30 14:57:56 CST 2024] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.83898391.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table changefeed_pause_resume.t1 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2129e07c0004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-mzktr, pid:6250, start at 2024-05-30 14:57:54.722255146 +0800 CST m=+5.856058255 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:54.730 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:54.719 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:54.719 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2129e07c0004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-mzktr, pid:6250, start at 2024-05-30 14:57:54.722255146 +0800 CST m=+5.856058255 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:54.730 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:54.719 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:54.719 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2129e11c0015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-mzktr, pid:6326, start at 2024-05-30 14:57:54.806373316 +0800 CST m=+5.868530774 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:54.814 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:54.810 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:54.810 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff successfully capture_id: deb04e05-92c9-47c8-b5ce-55866dd224cf table ddl_manager.finish_mark not exists for 19-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release wait process cdc.test exit for 1-th time... table changefeed_pause_resume.t1 exists table changefeed_pause_resume.t2 not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/api_v2 Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... [Thu May 30 14:57:59 CST 2024] <<<<<< START cdc server in processor_stop_delay case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorStopDelay=1*sleep(10000)' + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_stop_delay.76347636.out server --log-file /tmp/tidb_cdc_test/processor_stop_delay/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_stop_delay/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 cdc.test: no process found wait process cdc.test exit for 4-th time... table ddl_manager.finish_mark not exists for 20-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:59 GMT < Content-Length: 867 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/98d5b9ac-09d4-4747-9bb9-7aab49ce67e6 {"id":"98d5b9ac-09d4-4747-9bb9-7aab49ce67e6","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/e95abbc6-5001-4f8c-896a-f50ad7085d4b {"id":"e95abbc6-5001-4f8c-896a-f50ad7085d4b","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a319525 e95abbc6-5001-4f8c-896a-f50ad7085d4b /tidb/cdc/default/__cdc_meta__/owner/22318fc84a319552 98d5b9ac-09d4-4747-9bb9-7aab49ce67e6 /tidb/cdc/default/default/upstream/7374683301044003426 {"id":7374683301044003426,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/98d5b9ac-09d4-4747-9bb9-7aab49ce67e6 {"id":"98d5b9ac-09d4-4747-9bb9-7aab49ce67e6","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/e95abbc6-5001-4f8c-896a-f50ad7085d4b {"id":"e95abbc6-5001-4f8c-896a-f50ad7085d4b","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a319525 e95abbc6-5001-4f8c-896a-f50ad7085d4b /tidb/cdc/default/__cdc_meta__/owner/22318fc84a319552 98d5b9ac-09d4-4747-9bb9-7aab49ce67e6 /tidb/cdc/default/default/upstream/7374683301044003426 {"id":7374683301044003426,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/98d5b9ac-09d4-4747-9bb9-7aab49ce67e6 {"id":"98d5b9ac-09d4-4747-9bb9-7aab49ce67e6","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/e95abbc6-5001-4f8c-896a-f50ad7085d4b {"id":"e95abbc6-5001-4f8c-896a-f50ad7085d4b","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a319525 e95abbc6-5001-4f8c-896a-f50ad7085d4b /tidb/cdc/default/__cdc_meta__/owner/22318fc84a319552 98d5b9ac-09d4-4747-9bb9-7aab49ce67e6 /tidb/cdc/default/default/upstream/7374683301044003426 {"id":7374683301044003426,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Thu May 30 14:57:59 CST 2024] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.84348436.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc3.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data3 --cluster-id default --addr 127.0.0.1:8303 ++ curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info * About to connect() to 127.0.0.1 port 8303 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8303; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/fe1cd9c3-1823-4c69-83ea-1db1e01e0218 + [[ /tidb/cdc/default/__cdc_meta__/capture/fe1cd9c3-1823-4c69-83ea-1db1e01e0218 =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/fe1cd9c3-1823-4c69-83ea-1db1e01e0218' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/fe1cd9c3-1823-4c69-83ea-1db1e01e0218 + echo 'check failed' check failed + exit 1 run task failed 3-th time, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) process cdc.test already exit [Thu May 30 14:57:59 CST 2024] <<<<<< run test case new_ci_collation success! >>>>>> 0 [Thu May 30 14:58:00 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.36293631.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_owner_cleanup_stale_tasks.server3.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_owner_cleanup_stale_tasks.server3 --cluster-id default --addr 127.0.0.1:8302 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table changefeed_pause_resume.t2 exists table changefeed_pause_resume.t3 not exists for 1-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:02 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4792f11a-786c-4d8c-bfce-987b4dde2a98 {"id":"4792f11a-786c-4d8c-bfce-987b4dde2a98","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a496bba 4792f11a-786c-4d8c-bfce-987b4dde2a98 /tidb/cdc/default/default/upstream/7374683323834961520 {"id":7374683323834961520,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4792f11a-786c-4d8c-bfce-987b4dde2a98 {"id":"4792f11a-786c-4d8c-bfce-987b4dde2a98","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a496bba 4792f11a-786c-4d8c-bfce-987b4dde2a98 /tidb/cdc/default/default/upstream/7374683323834961520 {"id":7374683323834961520,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4792f11a-786c-4d8c-bfce-987b4dde2a98 {"id":"4792f11a-786c-4d8c-bfce-987b4dde2a98","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a496bba 4792f11a-786c-4d8c-bfce-987b4dde2a98 /tidb/cdc/default/default/upstream/7374683323834961520 {"id":7374683323834961520,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc) 3309751296 bytes in 36.67 secs (90269617 bytes/sec) [Pipeline] { [Pipeline] cache table changefeed_pause_resume.t3 exists table ddl_manager.finish_mark not exists for 21-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info * About to connect() to 127.0.0.1 port 8303 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8303 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8303 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:02 GMT < Content-Length: 1144 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1e61c353-5f82-4e45-88f9-353d61557b57 {"id":"1e61c353-5f82-4e45-88f9-353d61557b57","address":"127.0.0.1:8303","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/98d5b9ac-09d4-4747-9bb9-7aab49ce67e6 {"id":"98d5b9ac-09d4-4747-9bb9-7aab49ce67e6","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/e95abbc6-5001-4f8c-896a-f50ad7085d4b {"id":"e95abbc6-5001-4f8c-896a-f50ad7085d4b","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a319525 e95abbc6-5001-4f8c-896a-f50ad7085d4b /tidb/cdc/default/__cdc_meta__/owner/22318fc84a319552 98d5b9ac-09d4-4747-9bb9-7aab49ce67e6 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a319568 1e61c353-5f82-4e45-88f9-353d61557b57 /tidb/cdc/default/default/upstream/7374683301044003426 {"id":7374683301044003426,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1e61c353-5f82-4e45-88f9-353d61557b57 {"id":"1e61c353-5f82-4e45-88f9-353d61557b57","address":"127.0.0.1:8303","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/98d5b9ac-09d4-4747-9bb9-7aab49ce67e6 {"id":"98d5b9ac-09d4-4747-9bb9-7aab49ce67e6","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/e95abbc6-5001-4f8c-896a-f50ad7085d4b {"id":"e95abbc6-5001-4f8c-896a-f50ad7085d4b","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a319525 e95abbc6-5001-4f8c-896a-f50ad7085d4b /tidb/cdc/default/__cdc_meta__/owner/22318fc84a319552 98d5b9ac-09d4-4747-9bb9-7aab49ce67e6 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a319568 1e61c353-5f82-4e45-88f9-353d61557b57 /tidb/cdc/default/default/upstream/7374683301044003426 {"id":7374683301044003426,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1e61c353-5f82-4e45-88f9-353d61557b57 {"id":"1e61c353-5f82-4e45-88f9-353d61557b57","address":"127.0.0.1:8303","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/98d5b9ac-09d4-4747-9bb9-7aab49ce67e6 {"id":"98d5b9ac-09d4-4747-9bb9-7aab49ce67e6","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/e95abbc6-5001-4f8c-896a-f50ad7085d4b {"id":"e95abbc6-5001-4f8c-896a-f50ad7085d4b","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a319525 e95abbc6-5001-4f8c-896a-f50ad7085d4b /tidb/cdc/default/__cdc_meta__/owner/22318fc84a319552 98d5b9ac-09d4-4747-9bb9-7aab49ce67e6 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a319568 1e61c353-5f82-4e45-88f9-353d61557b57 /tidb/cdc/default/default/upstream/7374683301044003426 {"id":7374683301044003426,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.cli.8485.out cli changefeed create --start-ts=450114950519586817 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --server=127.0.0.1:8301 Create changefeed successfully! ID: 89e39a57-c33e-4427-90c6-ccacf2e07a3a Info: {"upstream_id":7374683301044003426,"namespace":"default","id":"89e39a57-c33e-4427-90c6-ccacf2e07a3a","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T14:58:02.96806446+08:00","start_ts":450114950519586817,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114950519586817,"checkpoint_ts":450114950519586817,"checkpoint_time":"2024-05-30 14:57:50.964"} PASS ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) coverage: 2.4% of statements in github.com/pingcap/tiflow/... Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:03 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00369d830 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114951897939974 MinTableBarrierTs:450114951897939974 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3e494e64-9d1e-4fd3-a49c-ae046c6dc99f {"id":"3e494e64-9d1e-4fd3-a49c-ae046c6dc99f","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/3efaf4bd-8e70-4182-bd87-02fec40a23fc {"id":"3efaf4bd-8e70-4182-bd87-02fec40a23fc","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/deb04e05-92c9-47c8-b5ce-55866dd224cf {"id":"deb04e05-92c9-47c8-b5ce-55866dd224cf","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525aaf 3efaf4bd-8e70-4182-bd87-02fec40a23fc /tidb/cdc/default/__cdc_meta__/owner/22318fc849525af1 deb04e05-92c9-47c8-b5ce-55866dd224cf /tidb/cdc/default/__cdc_meta__/owner/22318fc849525b55 3e494e64-9d1e-4fd3-a49c-ae046c6dc99f /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114951897939974,"min-table-barrier-ts":450114951897939974,"admin-job-type":0} /tidb/cdc/default/default/task/position/3e494e64-9d1e-4fd3-a49c-ae046c6dc99f/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/3efaf4bd-8e70-4182-bd87-02fec40a23fc/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/deb04e05-92c9-47c8-b5ce-55866dd224cf/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00369d830 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114951897939974 MinTableBarrierTs:450114951897939974 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3e494e64-9d1e-4fd3-a49c-ae046c6dc99f {"id":"3e494e64-9d1e-4fd3-a49c-ae046c6dc99f","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/3efaf4bd-8e70-4182-bd87-02fec40a23fc {"id":"3efaf4bd-8e70-4182-bd87-02fec40a23fc","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/deb04e05-92c9-47c8-b5ce-55866dd224cf {"id":"deb04e05-92c9-47c8-b5ce-55866dd224cf","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525aaf 3efaf4bd-8e70-4182-bd87-02fec40a23fc /tidb/cdc/default/__cdc_meta__/owner/22318fc849525af1 deb04e05-92c9-47c8-b5ce-55866dd224cf /tidb/cdc/default/__cdc_meta__/owner/22318fc849525b55 3e494e64-9d1e-4fd3-a49c-ae046c6dc99f /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114951897939974,"min-table-barrier-ts":450114951897939974,"admin-job-type":0} /tidb/cdc/default/default/task/position/3e494e64-9d1e-4fd3-a49c-ae046c6dc99f/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/3efaf4bd-8e70-4182-bd87-02fec40a23fc/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/deb04e05-92c9-47c8-b5ce-55866dd224cf/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00369d830 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114951897939974 MinTableBarrierTs:450114951897939974 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3e494e64-9d1e-4fd3-a49c-ae046c6dc99f {"id":"3e494e64-9d1e-4fd3-a49c-ae046c6dc99f","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/3efaf4bd-8e70-4182-bd87-02fec40a23fc {"id":"3efaf4bd-8e70-4182-bd87-02fec40a23fc","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/deb04e05-92c9-47c8-b5ce-55866dd224cf {"id":"deb04e05-92c9-47c8-b5ce-55866dd224cf","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525aaf 3efaf4bd-8e70-4182-bd87-02fec40a23fc /tidb/cdc/default/__cdc_meta__/owner/22318fc849525af1 deb04e05-92c9-47c8-b5ce-55866dd224cf /tidb/cdc/default/__cdc_meta__/owner/22318fc849525b55 3e494e64-9d1e-4fd3-a49c-ae046c6dc99f /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114951897939974,"min-table-barrier-ts":450114951897939974,"admin-job-type":0} /tidb/cdc/default/default/task/position/3e494e64-9d1e-4fd3-a49c-ae046c6dc99f/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/3efaf4bd-8e70-4182-bd87-02fec40a23fc/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/deb04e05-92c9-47c8-b5ce-55866dd224cf/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list --server 'http://127.0.0.1:8302' 2>&1 | grep '"is-owner": true' run task failed 1-th time, retry later table processor_stop_delay.t not exists for 1-th check, retry later + set +x table multi_capture_1.usertable not exists for 1-th check, retry later check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 22-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test cli capture list --server 'http://127.0.0.1:8302' 2>&1 | grep '"is-owner": true' table processor_stop_delay.t exists check diff failed 1-th time, retry later run task failed 2-th time, retry later check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info= + [[ '' =~ capture ]] + echo 'check pass' check pass + exit 0 run task successfully check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Thu May 30 14:58:05 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/NewChangefeedRetryError=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.56345636.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table multi_capture_1.usertable exists table multi_capture_2.usertable not exists for 1-th check, retry later check diff failed 2-th time, retry later table ddl_manager.finish_mark not exists for 23-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212a81b40019 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3, pid:4154, start at 2024-05-30 14:58:05.072974139 +0800 CST m=+5.439572050 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:05.080 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:05.037 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:05.037 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212a81b40019 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3, pid:4154, start at 2024-05-30 14:58:05.072974139 +0800 CST m=+5.439572050 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:05.080 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:05.037 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:05.037 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212a83300013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3, pid:4226, start at 2024-05-30 14:58:05.15842365 +0800 CST m=+5.431778308 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:05.165 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:05.132 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:05.132 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_replicate_gbk/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_replicate_gbk/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/consistent_replicate_gbk/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/consistent_replicate_gbk/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_replicate_gbk/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully check diff failed 3-th time, retry later table ddl_manager.finish_mark not exists for 24-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:08 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ee1626c9-9e74-4d74-9f42-045991e4ee8b {"id":"ee1626c9-9e74-4d74-9f42-045991e4ee8b","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318fc849f528a4 ee1626c9-9e74-4d74-9f42-045991e4ee8b /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7374683233227573581,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T14:57:45.386750116+08:00","start-ts":450114947678470148,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"pending","error":{"time":"2024-05-30T14:58:06.151290505+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114954494214150} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":450114950837305351,"min-table-barrier-ts":450114951007698950,"admin-job-type":1} /tidb/cdc/default/default/upstream/7374683233227573581 {"id":7374683233227573581,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ee1626c9-9e74-4d74-9f42-045991e4ee8b {"id":"ee1626c9-9e74-4d74-9f42-045991e4ee8b","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318fc849f528a4 ee1626c9-9e74-4d74-9f42-045991e4ee8b /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7374683233227573581,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T14:57:45.386750116+08:00","start-ts":450114947678470148,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"pending","error":{"time":"2024-05-30T14:58:06.151290505+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114954494214150} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":450114950837305351,"min-table-barrier-ts":450114951007698950,"admin-job-type":1} /tidb/cdc/default/default/upstream/7374683233227573581 {"id":7374683233227573581,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ee1626c9-9e74-4d74-9f42-045991e4ee8b {"id":"ee1626c9-9e74-4d74-9f42-045991e4ee8b","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318fc849f528a4 ee1626c9-9e74-4d74-9f42-045991e4ee8b /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7374683233227573581,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T14:57:45.386750116+08:00","start-ts":450114947678470148,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"pending","error":{"time":"2024-05-30T14:58:06.151290505+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114954494214150} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":450114950837305351,"min-table-barrier-ts":450114951007698950,"admin-job-type":1} /tidb/cdc/default/default/upstream/7374683233227573581 {"id":7374683233227573581,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-error warning failpoint injected retriable error + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error + expected_state=warning + error_msg=failpoint + tls_dir=error + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error -s table multi_capture_2.usertable exists table multi_capture_3.usertable not exists for 1-th check, retry later + info='{ "upstream_id": 7374683233227573581, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 450114950837305351, "checkpoint_time": "2024-05-30 14:57:52.176", "error": { "time": "2024-05-30T14:58:06.151290505+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } }' + echo '{ "upstream_id": 7374683233227573581, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 450114950837305351, "checkpoint_time": "2024-05-30 14:57:52.176", "error": { "time": "2024-05-30T14:58:06.151290505+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } }' { "upstream_id": 7374683233227573581, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 450114950837305351, "checkpoint_time": "2024-05-30 14:57:52.176", "error": { "time": "2024-05-30T14:58:06.151290505+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } } ++ echo '{' '"upstream_id":' 7374683233227573581, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"warning",' '"checkpoint_tso":' 450114950837305351, '"checkpoint_time":' '"2024-05-30' '14:57:52.176",' '"error":' '{' '"time":' '"2024-05-30T14:58:06.151290505+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrOwnerUnknown",' '"message":' '"failpoint' injected retriable 'error"' '}' '}' ++ jq -r .state + state=warning + [[ ! warning == \w\a\r\n\i\n\g ]] ++ echo '{' '"upstream_id":' 7374683233227573581, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"warning",' '"checkpoint_tso":' 450114950837305351, '"checkpoint_time":' '"2024-05-30' '14:57:52.176",' '"error":' '{' '"time":' '"2024-05-30T14:58:06.151290505+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrOwnerUnknown",' '"message":' '"failpoint' injected retriable 'error"' '}' '}' ++ jq -r .error.message + message='failpoint injected retriable error' + [[ ! failpoint injected retriable error =~ failpoint ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5725.out cli changefeed remove -c changefeed-error VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212ab8c0000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-js3jc, pid:6972, start at 2024-05-30 14:58:08.575213043 +0800 CST m=+5.499533413 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:08.583 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:08.560 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:08.560 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Thu May 30 14:58:08 CST 2024] <<<<<< START cdc server in consistent_replicate_gbk case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_gbk.55365538.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_gbk/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_gbk/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 1-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/batch_add_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... Changefeed remove successfully. ID: changefeed-error CheckpointTs: 450114950837305351 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... cdc.test cli capture list --server 'http://127.0.0.1:8302' 2>&1 | grep '"is-owner": true' table multi_capture_3.usertable exists table multi_capture_4.usertable not exists for 1-th check, retry later check diff failed 4-th time, retry later table ddl_manager.finish_mark not exists for 25-th check, retry later + set +x check_no_changefeed 127.0.0.1:2379 parse error: Invalid numeric literal at line 1, column 6 run task successfully "is-owner": true, run task successfully nonempty select id, val from test.availability1 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212ab8c0000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-js3jc, pid:6972, start at 2024-05-30 14:58:08.575213043 +0800 CST m=+5.499533413 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:08.583 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:08.560 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:08.560 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212abaf8000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-js3jc, pid:7050, start at 2024-05-30 14:58:08.713607779 +0800 CST m=+5.574323130 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:08.721 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:08.702 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:08.702 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/api_v2/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/api_v2/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/api_v2/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/api_v2/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/api_v2/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 1-th time... check diff failed 2-th time, retry later wait process cdc.test exit for 2-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:11 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cf7a09ae-6bfa-40df-8a30-62507b2cb34f {"id":"cf7a09ae-6bfa-40df-8a30-62507b2cb34f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a7763b9 cf7a09ae-6bfa-40df-8a30-62507b2cb34f /tidb/cdc/default/default/upstream/7374683371390909134 {"id":7374683371390909134,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cf7a09ae-6bfa-40df-8a30-62507b2cb34f {"id":"cf7a09ae-6bfa-40df-8a30-62507b2cb34f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a7763b9 cf7a09ae-6bfa-40df-8a30-62507b2cb34f /tidb/cdc/default/default/upstream/7374683371390909134 {"id":7374683371390909134,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cf7a09ae-6bfa-40df-8a30-62507b2cb34f {"id":"cf7a09ae-6bfa-40df-8a30-62507b2cb34f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a7763b9 cf7a09ae-6bfa-40df-8a30-62507b2cb34f /tidb/cdc/default/default/upstream/7374683371390909134 {"id":7374683371390909134,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only table multi_capture_4.usertable exists check diff failed 1-th time, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc) 3309751296 bytes in 7.94 secs (416589931 bytes/sec) [Pipeline] { [Pipeline] cache check diff failed 5-th time, retry later + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Thu May 30 14:58:12 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/InjectChangefeedDDLError=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.58115813.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 26-th check, retry later nonempty select id, val from test.availability1 where id=1 and val=1 run task successfully nonempty select id, val from test.availability1 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later table consistent_replicate_gbk.GBKTABLE not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/batch_add_table Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... check diff failed 3-th time, retry later [Thu May 30 14:58:13 CST 2024] <<<<<< START cdc server in api_v2 case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.api_v2.84038405.out server --log-file /tmp/tidb_cdc_test/api_v2/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/api_v2/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 27-th check, retry later check diff successfully ***************** properties ***************** "recordcount"="20" "mysql.host"="127.0.0.1" "dotransactions"="false" "mysql.port"="4000" "readallfields"="true" "insertproportion"="0" "requestdistribution"="uniform" "threadcount"="2" "workload"="core" "readproportion"="0" "mysql.user"="root" "operationcount"="0" "updateproportion"="0" "scanproportion"="0" "mysql.db"="multi_capture_1" ********************************************** Run finished, takes 11.674861ms INSERT - Takes(s): 0.0, Count: 20, OPS: 1926.5, Avg(us): 1083, Min(us): 504, Max(us): 2259, 95th(us): 2000, 99th(us): 3000 ***************** properties ***************** "readallfields"="true" "threadcount"="2" "recordcount"="20" "dotransactions"="false" "operationcount"="0" "updateproportion"="0" "readproportion"="0" "workload"="core" "mysql.host"="127.0.0.1" "mysql.db"="multi_capture_2" "mysql.port"="4000" "scanproportion"="0" "mysql.user"="root" "requestdistribution"="uniform" "insertproportion"="0" ********************************************** Run finished, takes 12.581497ms INSERT - Takes(s): 0.0, Count: 20, OPS: 1753.9, Avg(us): 1052, Min(us): 433, Max(us): 1891, 95th(us): 2000, 99th(us): 2000 ***************** properties ***************** "mysql.host"="127.0.0.1" "mysql.db"="multi_capture_3" "dotransactions"="false" "updateproportion"="0" "threadcount"="2" "readallfields"="true" "requestdistribution"="uniform" "mysql.user"="root" "operationcount"="0" "insertproportion"="0" "workload"="core" "readproportion"="0" "recordcount"="20" "mysql.port"="4000" "scanproportion"="0" ********************************************** Run finished, takes 11.53587ms INSERT - Takes(s): 0.0, Count: 20, OPS: 1927.4, Avg(us): 992, Min(us): 502, Max(us): 1833, 95th(us): 2000, 99th(us): 2000 ***************** properties ***************** "mysql.db"="multi_capture_4" "dotransactions"="false" "threadcount"="2" "updateproportion"="0" "insertproportion"="0" "mysql.port"="4000" "mysql.host"="127.0.0.1" "mysql.user"="root" "scanproportion"="0" "operationcount"="0" "workload"="core" "readproportion"="0" "readallfields"="true" "requestdistribution"="uniform" "recordcount"="20" ********************************************** Run finished, takes 21.703504ms INSERT - Takes(s): 0.0, Count: 20, OPS: 1521.5, Avg(us): 2109, Min(us): 447, Max(us): 8468, 95th(us): 9000, 99th(us): 9000 check diff failed 1-th time, retry later check diff successfully nonempty select id, val from test.availability1 where id=1 and val=22 run task successfully empty select id, val from test.availability1 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later table consistent_replicate_gbk.GBKTABLE exists table consistent_replicate_gbk.check1 not exists for 1-th check, retry later check diff failed 4-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:15 GMT < Content-Length: 681 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c2d84c33-c7c3-43bd-a9dc-3d507d184ae8 {"id":"c2d84c33-c7c3-43bd-a9dc-3d507d184ae8","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318fc849f528f8 c2d84c33-c7c3-43bd-a9dc-3d507d184ae8 /tidb/cdc/default/default/upstream/7374683233227573581 {"id":7374683233227573581,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c2d84c33-c7c3-43bd-a9dc-3d507d184ae8 {"id":"c2d84c33-c7c3-43bd-a9dc-3d507d184ae8","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318fc849f528f8 c2d84c33-c7c3-43bd-a9dc-3d507d184ae8 /tidb/cdc/default/default/upstream/7374683233227573581 {"id":7374683233227573581,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c2d84c33-c7c3-43bd-a9dc-3d507d184ae8 {"id":"c2d84c33-c7c3-43bd-a9dc-3d507d184ae8","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318fc849f528f8 c2d84c33-c7c3-43bd-a9dc-3d507d184ae8 /tidb/cdc/default/default/upstream/7374683233227573581 {"id":7374683233227573581,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5856.out cli changefeed create --start-ts=450114947678470148 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' -c changefeed-error-1 Create changefeed successfully! ID: changefeed-error-1 Info: {"upstream_id":7374683233227573581,"namespace":"default","id":"changefeed-error-1","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T14:58:16.285421359+08:00","start_ts":450114947678470148,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114947678470148,"checkpoint_ts":450114947678470148,"checkpoint_time":"2024-05-30 14:57:40.126"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 28-th check, retry later < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:16 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1b2fac6e-e3eb-4e8a-970a-7806832ef7b1 {"id":"1b2fac6e-e3eb-4e8a-970a-7806832ef7b1","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a84c1c7 1b2fac6e-e3eb-4e8a-970a-7806832ef7b1 /tidb/cdc/default/default/upstream/7374683386590487247 {"id":7374683386590487247,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1b2fac6e-e3eb-4e8a-970a-7806832ef7b1 {"id":"1b2fac6e-e3eb-4e8a-970a-7806832ef7b1","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a84c1c7 1b2fac6e-e3eb-4e8a-970a-7806832ef7b1 /tidb/cdc/default/default/upstream/7374683386590487247 {"id":7374683386590487247,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1b2fac6e-e3eb-4e8a-970a-7806832ef7b1 {"id":"1b2fac6e-e3eb-4e8a-970a-7806832ef7b1","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a84c1c7 1b2fac6e-e3eb-4e8a-970a-7806832ef7b1 /tidb/cdc/default/default/upstream/7374683386590487247 {"id":7374683386590487247,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: cf-blackhole Info: {"upstream_id":7374683386590487247,"namespace":"default","id":"cf-blackhole","sink_uri":"blackhole:","create_time":"2024-05-30T14:58:16.930117049+08:00","start_ts":450114957294960644,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114957294960644,"checkpoint_ts":450114957294960644,"checkpoint_time":"2024-05-30 14:58:16.810"} + endpoints=http://127.0.0.1:2379 + changefeed_id=cf-blackhole + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c cf-blackhole -s + info='{ "upstream_id": 7374683386590487247, "namespace": "default", "id": "cf-blackhole", "state": "normal", "checkpoint_tso": 450114957294960644, "checkpoint_time": "2024-05-30 14:58:16.810", "error": null }' + echo '{ "upstream_id": 7374683386590487247, "namespace": "default", "id": "cf-blackhole", "state": "normal", "checkpoint_tso": 450114957294960644, "checkpoint_time": "2024-05-30 14:58:16.810", "error": null }' { "upstream_id": 7374683386590487247, "namespace": "default", "id": "cf-blackhole", "state": "normal", "checkpoint_tso": 450114957294960644, "checkpoint_time": "2024-05-30 14:58:16.810", "error": null } ++ echo '{' '"upstream_id":' 7374683386590487247, '"namespace":' '"default",' '"id":' '"cf-blackhole",' '"state":' '"normal",' '"checkpoint_tso":' 450114957294960644, '"checkpoint_time":' '"2024-05-30' '14:58:16.810",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374683386590487247, '"namespace":' '"default",' '"id":' '"cf-blackhole",' '"state":' '"normal",' '"checkpoint_tso":' 450114957294960644, '"checkpoint_time":' '"2024-05-30' '14:58:16.810",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + GO111MODULE=on + go run main.go model.go request.go cases.go go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f empty select id, val from test.availability1 where id=1 run task successfully test_owner_cleanup_stale_tasks pass go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20240527072219-a97e6464c01d go: downloading github.com/tikv/pd/client v0.0.0-20240210135946-3488a653ddd9 go: downloading github.com/gin-gonic/gin v1.9.1 go: downloading github.com/pingcap/kvproto v0.0.0-20240522024016-df42997c2c57 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading google.golang.org/grpc v1.64.0 go: downloading github.com/google/uuid v1.6.0 go: downloading golang.org/x/sync v0.7.0 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading golang.org/x/time v0.5.0 go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20240527072219-a97e6464c01d go: downloading github.com/pingcap/failpoint v0.0.0-20240412033321-fd0796e60f86 go: downloading go.etcd.io/etcd/server/v3 v3.5.12 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20240510070134-e7f530c64790 go: downloading github.com/prometheus/client_golang v1.19.1 go: downloading github.com/apache/pulsar-client-go v0.11.0 go: downloading github.com/aws/aws-sdk-go-v2 v1.19.1 go: downloading github.com/tinylib/msgp v1.1.6 go: downloading github.com/IBM/sarama v1.41.2 go: downloading github.com/xdg/scram v1.0.5 go: downloading github.com/benbjohnson/clock v1.3.5 go: downloading github.com/uber-go/atomic v1.4.0 go: downloading go.uber.org/atomic v1.11.0 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/cockroachdb/pebble v0.0.0-20220415182917-06c9d3be25b3 go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.1 go: downloading github.com/KimMachineGun/automemlimit v0.2.4 go: downloading github.com/aws/aws-sdk-go v1.44.259 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/shirou/gopsutil/v3 v3.24.4 go: downloading github.com/klauspost/compress v1.17.8 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b check diff successfully wait process cdc.test exit for 1-th time... + set +x check_changefeed_status 127.0.0.1:8300 changefeed-error-1 warning last_warning ErrExecDDLFailed + endpoint=127.0.0.1:8300 + changefeed_id=changefeed-error-1 + expected_state=warning + field=last_warning + error_pattern=ErrExecDDLFailed ++ curl 127.0.0.1:8300/api/v2/changefeeds/changefeed-error-1/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed go: downloading github.com/cenkalti/backoff/v4 v4.2.1 go: downloading github.com/stretchr/testify v1.9.0 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading github.com/goccy/go-json v0.10.2 go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 86 100 86 0 0 588 0 --:--:-- --:--:-- --:--:-- 589 + info='{"state":"normal","resolved_ts":450114947678470148,"checkpoint_ts":450114947678470148}' + echo '{"state":"normal","resolved_ts":450114947678470148,"checkpoint_ts":450114947678470148}' {"state":"normal","resolved_ts":450114947678470148,"checkpoint_ts":450114947678470148} ++ echo '{"state":"normal","resolved_ts":450114947678470148,"checkpoint_ts":450114947678470148}' ++ jq -r .state + state=normal + [[ ! normal == \w\a\r\n\i\n\g ]] + echo 'changefeed state normal does not equal to warning' changefeed state normal does not equal to warning + exit 1 run task failed 1-th time, retry later go: downloading cloud.google.com/go v0.112.2 go: downloading github.com/google/btree v1.1.2 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... table consistent_replicate_gbk.check1 not exists for 2-th check, retry later go: downloading github.com/xdg/stringprep v1.0.3 go: downloading golang.org/x/crypto v0.23.0 check diff failed 5-th time, retry later table ddl_manager.finish_mark not exists for 29-th check, retry later go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading github.com/containerd/cgroups v1.0.4 go: downloading github.com/philhofer/fwd v1.1.1 go: downloading github.com/gin-contrib/sse v0.1.0 go: downloading golang.org/x/net v0.25.0 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/aws/smithy-go v1.13.5 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/prometheus/common v0.53.0 go: downloading github.com/prometheus/procfs v0.15.0 go: downloading google.golang.org/protobuf v1.34.1 go: downloading github.com/golang/protobuf v1.5.4 go: downloading github.com/pkg/errors v0.9.1 wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading github.com/gavv/monotime v0.0.0-20190418164738-30dba4353424 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading github.com/stretchr/objx v0.5.2 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/bits-and-blooms/bitset v1.4.0 go: downloading github.com/linkedin/goavro/v2 v2.11.1 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading go.etcd.io/etcd/pkg/v3 v3.5.12 go: downloading github.com/eapache/go-resiliency v1.4.0 go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 go: downloading github.com/eapache/queue v1.1.0 go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/jcmturner/gofork v1.7.6 go: downloading github.com/jcmturner/gokrb5/v8 v8.4.4 go: downloading github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 go: downloading golang.org/x/sys v0.20.0 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0 go: downloading github.com/soheilhy/cmux v0.1.5 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 go: downloading go.etcd.io/bbolt v1.3.9 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.22.0 go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading go.opentelemetry.io/otel/sdk v1.22.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 check diff failed 2-th time, retry later wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 14:58:19 CST 2024] <<<<<< run test case multi_capture success! >>>>>> go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading golang.org/x/oauth2 v0.18.0 go: downloading google.golang.org/api v0.170.0 Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore cdc.test: no process found wait process cdc.test exit for 4-th time... Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc) 3309751296 bytes in 4.92 secs (672337803 bytes/sec) [Pipeline] { [Pipeline] cache Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) process cdc.test already exit run test case test_owner_retryable_error [Thu May 30 14:58:19 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/capture/capture-campaign-compacted-error=1*return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.38643866.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_owner_retryable_error.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_owner_retryable_error.server1 --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo '' + '[' 0 -eq 50 ']' + sleep 3 check_changefeed_status 127.0.0.1:8300 changefeed-error-1 warning last_warning ErrExecDDLFailed + endpoint=127.0.0.1:8300 + changefeed_id=changefeed-error-1 + expected_state=warning + field=last_warning + error_pattern=ErrExecDDLFailed ++ curl 127.0.0.1:8300/api/v2/changefeeds/changefeed-error-1/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/go-mysql-org/go-mysql v1.7.1-0.20240507075657-2bd4573edde2 go: downloading github.com/cakturk/go-netstat v0.0.0-20200220111822-e5b49efee7a5 go: downloading golang.org/x/text v0.15.0 table consistent_replicate_gbk.check1 exists check diff successfully 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 253 100 253 0 0 2319 0 --:--:-- --:--:-- --:--:-- 2342 + info='{"state":"warning","resolved_ts":450114948071686162,"checkpoint_ts":450114948071686162,"last_warning":{"time":"2024-05-30T14:58:18.111317076+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}}' + echo '{"state":"warning","resolved_ts":450114948071686162,"checkpoint_ts":450114948071686162,"last_warning":{"time":"2024-05-30T14:58:18.111317076+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}}' {"state":"warning","resolved_ts":450114948071686162,"checkpoint_ts":450114948071686162,"last_warning":{"time":"2024-05-30T14:58:18.111317076+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}} ++ echo '{"state":"warning","resolved_ts":450114948071686162,"checkpoint_ts":450114948071686162,"last_warning":{"time":"2024-05-30T14:58:18.111317076+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec' DDL 'failed"}}' ++ jq -r .state + state=warning + [[ ! warning == \w\a\r\n\i\n\g ]] + [[ -z last_warning ]] ++ jq -r .last_warning.message ++ echo '{"state":"warning","resolved_ts":450114948071686162,"checkpoint_ts":450114948071686162,"last_warning":{"time":"2024-05-30T14:58:18.111317076+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec' DDL 'failed"}}' + error_msg='[CDC:ErrExecDDLFailed]exec DDL failed' + [[ ! [CDC:ErrExecDDLFailed]exec DDL failed =~ ErrExecDDLFailed ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5915.out cli changefeed remove -c changefeed-error-1 check diff failed 6-th time, retry later go: downloading github.com/pierrec/lz4 v2.6.1+incompatible wait process cdc.test exit for 1-th time... go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240521202816-d264139d666e go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/hashicorp/golang-lru v0.5.1 go: downloading github.com/segmentio/kafka-go v0.4.41-0.20230526171612-f057b1d369cd go: downloading github.com/imdario/mergo v0.3.16 go: downloading github.com/DATA-DOG/go-sqlmock v1.5.0 go: downloading github.com/go-playground/validator/v10 v10.14.0 go: downloading github.com/pelletier/go-toml/v2 v2.0.8 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/godbus/dbus/v5 v5.0.4 go: downloading github.com/opencontainers/runtime-spec v1.0.2 go: downloading github.com/ugorji/go/codec v1.2.11 go: downloading github.com/cilium/ebpf v0.4.0 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.1 go: downloading github.com/AthenZ/athenz v1.10.39 go: downloading github.com/spaolacci/murmur3 v1.1.0 go: downloading golang.org/x/mod v0.17.0 go: downloading github.com/DataDog/zstd v1.5.5 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/hashicorp/errwrap v1.0.0 go: downloading github.com/jcmturner/dnsutils/v2 v2.0.0 go: downloading github.com/hashicorp/go-uuid v1.0.3 go: downloading github.com/spf13/pflag v1.0.5 go: downloading go.etcd.io/etcd/raft/v3 v3.5.12 go: downloading github.com/jonboulle/clockwork v0.4.0 go: downloading github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 go: downloading github.com/gorilla/websocket v1.5.1 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading go.opentelemetry.io/proto/otlp v1.1.0 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading github.com/Masterminds/semver v1.5.0 go: downloading github.com/siddontang/go-log v0.0.0-20180807004314-8d05993dda07 go: downloading github.com/siddontang/go v0.0.0-20180604090527-bdc77568d726 go: downloading github.com/shopspring/decimal v1.3.0 go: downloading github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 go: downloading github.com/xdg-go/scram v1.1.2 go: downloading github.com/pingcap/check v0.0.0-20211026125417-57bd13f7b5f0 check diff failed 3-th time, retry later table ddl_manager.finish_mark not exists for 30-th check, retry later Changefeed remove successfully. ID: changefeed-error-1 CheckpointTs: 450114948071686162 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... wait process cdc.test exit for 2-th time... go: downloading github.com/pingcap/tipb v0.0.0-20240507090649-2bf6bb0cb996 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.6 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/BurntSushi/toml v1.3.2 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/99designs/keyring v1.2.1 go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible go: downloading github.com/jcmturner/rpc/v2 v2.0.3 go: downloading github.com/golang-jwt/jwt/v4 v4.5.0 go: downloading go.etcd.io/etcd/client/v2 v2.305.12 go: downloading github.com/go-logr/logr v1.4.1 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/aws/aws-sdk-go-v2/config v1.18.30 go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.13.29 go: downloading github.com/aws/aws-sdk-go-v2/service/glue v1.58.1 go: downloading github.com/jarcoal/httpmock v1.2.0 go: downloading github.com/jcmturner/aescts/v2 v2.0.0 go: downloading github.com/mailru/easyjson v0.7.7 go: downloading github.com/blacktear23/go-proxyprotocol v1.0.6 go: downloading github.com/gorilla/mux v1.8.0 go: downloading github.com/pingcap/fn v1.0.0 go: downloading github.com/tiancaiamao/appdash v0.0.0-20181126055449-889f96f722a2 go: downloading github.com/sourcegraph/appdash-data v0.0.0-20151005221446-73f23eafcf67 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 go: downloading github.com/xdg-go/pbkdf2 v1.0.0 go: downloading github.com/xdg-go/stringprep v1.0.4 go: downloading github.com/go-playground/universal-translator v0.18.1 go: downloading github.com/leodido/go-urn v1.2.4 go: downloading github.com/gabriel-vasile/mimetype v1.4.2 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/danjacques/gofslock v0.0.0-20220131014315-6e321f4509c8 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.1 go: downloading github.com/iancoleman/strcase v0.2.0 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/dvsekhvalnov/jose2go v1.5.0 go: downloading github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c go: downloading github.com/mtibben/percent v0.2.1 go: downloading golang.org/x/term v0.20.0 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.0 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/sasha-s/go-deadlock v0.3.1 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.6 go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.3.37 go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.12.14 go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.14 go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.20.1 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/go-playground/locales v0.14.1 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/fatih/color v1.16.0 go: downloading github.com/vbauerster/mpb/v7 v7.5.3 go: downloading github.com/spf13/cobra v1.8.0 go: downloading github.com/pingcap/badger v1.5.1-0.20230103063557-828f39b09b6d go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-runewidth v0.0.15 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/petermattis/goid v0.0.0-20231207134359-e60b3f734c67 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.36 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.30 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading k8s.io/api v0.28.6 go: downloading github.com/emirpasic/gods v1.18.1 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.12.0 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/golang-jwt/jwt/v5 v5.2.0 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.30 go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d go: downloading github.com/lestrrat-go/httpcc v1.0.1 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 14:58:21 CST 2024] <<<<<< START cdc server in consistent_replicate_gbk case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_gbk.57475749.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_gbk/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_gbk/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 check diff successfully + set +x wait process cdc.test exit for 1-th time... check diff failed 4-th time, retry later go: downloading k8s.io/apimachinery v0.28.6 table ddl_manager.finish_mark not exists for 31-th check, retry later wait process cdc.test exit for 1-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:22 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004494990 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114957402701833 MinTableBarrierTs:450114958713421830 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114957402701833, checkpointTs: 450114957402701833, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114957402701833, checkpointTs: 450114957402701833, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114957402701833, checkpointTs: 450114957402701833, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/12ce2e67-c501-43d7-a518-4cd74ef17420 {"id":"12ce2e67-c501-43d7-a518-4cd74ef17420","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525bef 12ce2e67-c501-43d7-a518-4cd74ef17420 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114957402701833,"min-table-barrier-ts":450114958713421830,"admin-job-type":0} /tidb/cdc/default/default/task/position/12ce2e67-c501-43d7-a518-4cd74ef17420/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004494990 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114957402701833 MinTableBarrierTs:450114958713421830 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114957402701833, checkpointTs: 450114957402701833, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114957402701833, checkpointTs: 450114957402701833, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114957402701833, checkpointTs: 450114957402701833, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/12ce2e67-c501-43d7-a518-4cd74ef17420 {"id":"12ce2e67-c501-43d7-a518-4cd74ef17420","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525bef 12ce2e67-c501-43d7-a518-4cd74ef17420 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114957402701833,"min-table-barrier-ts":450114958713421830,"admin-job-type":0} /tidb/cdc/default/default/task/position/12ce2e67-c501-43d7-a518-4cd74ef17420/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/+ grep -q 'failed to get info:' default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004494990 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114957402701833 MinTableBarrierTs:450114958713421830 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114957402701833, checkpointTs: 450114957402701833, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114957402701833, checkpointTs: 450114957402701833, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114957402701833, checkpointTs: 450114957402701833, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/12ce2e67-c501-43d7-a518-4cd74ef17420 {"id":"12ce2e67-c501-43d7-a518-4cd74ef17420","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525bef 12ce2e67-c501-43d7-a518-4cd74ef17420 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114957402701833,"min-table-barrier-ts":450114958713421830,"admin-job-type":0} /tidb/cdc/default/default/task/position/12ce2e67-c501-43d7-a518-4cd74ef17420/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' wait process cdc.test exit for 2-th time... go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading k8s.io/klog/v2 v2.120.1 go: downloading github.com/google/gofuzz v1.2.0 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... go: downloading github.com/jmespath/go-jmespath v0.4.0 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Thu May 30 14:58:23 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/txnutil/gc/InjectActualGCSafePoint=return(9223372036854775807)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.59645966.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info go: downloading github.com/ardielle/ardielle-go v1.5.2 * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 14:58:24 CST 2024] <<<<<< run test case processor_stop_delay success! >>>>>> "is-owner": true, run task successfully table ddl_manager.finish_mark not exists for 32-th check, retry later go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading github.com/google/s2a-go v0.1.7 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading github.com/felixge/httpsnoop v1.0.4 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:24 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/10c33d43-351b-4b6e-9ac7-b39ed4bd0736 {UpstreamID:7374683371390909134 Namespace:default ID:10c33d43-351b-4b6e-9ac7-b39ed4bd0736 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:58:12.154676596 +0800 CST StartTs:450114956044009477 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00403acf0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114956069961733} {CheckpointTs:450114959071772674 MinTableBarrierTs:450114959071772674 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114959071772675, checkpointTs: 450114959071772674, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114959071772675, checkpointTs: 450114959071772674, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a4eb1fdd-ccc1-4409-bde8-a7adeb529ab7 {"id":"a4eb1fdd-ccc1-4409-bde8-a7adeb529ab7","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a77645e a4eb1fdd-ccc1-4409-bde8-a7adeb529ab7 /tidb/cdc/default/default/changefeed/info/10c33d43-351b-4b6e-9ac7-b39ed4bd0736 {"upstream-id":7374683371390909134,"namespace":"default","changefeed-id":"10c33d43-351b-4b6e-9ac7-b39ed4bd0736","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:58:12.154676596+08:00","start-ts":450114956044009477,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114956069961733} /tidb/cdc/default/default/changefeed/status/10c33d43-351b-4b6e-9ac7-b39ed4bd0736 {"checkpoint-ts":450114959071772674,"min-table-barrier-ts":450114959097724934,"admin-job-type":0} /tidb/cdc/default/default/task/position/a4eb1fdd-ccc1-4409-bde8-a7adeb529ab7/10c33d43-351b-4b6e-9ac7-b39ed4bd0736 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683371390909134 {"id":7374683371390909134,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/10c33d43-351b-4b6e-9ac7-b39ed4bd0736 {UpstreamID:7374683371390909134 Namespace:default ID:10c33d43-351b-4b6e-9ac7-b39ed4bd0736 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:58:12.154676596 +0800 CST StartTs:450114956044009477 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00403acf0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114956069961733} {CheckpointTs:450114959071772674 MinTableBarrierTs:450114959071772674 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114959071772675, checkpointTs: 450114959071772674, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114959071772675, checkpointTs: 450114959071772674, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a4eb1fdd-ccc1-4409-bde8-a7adeb529ab7 {"id":"a4eb1fdd-ccc1-4409-bde8-a7adeb529ab7","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a77645e a4eb1fdd-ccc1-4409-bde8-a7adeb529ab7 /tidb/cdc/default/default/changefeed/info/10c33d43-351b-4b6e-9ac7-b39ed4bd0736 {"upstream-id":7374683371390909134,"namespace":"default","changefeed-id":"10c33d43-351b-4b6e-9ac7-b39ed4bd0736","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:58:12.154676596+08:00","start-ts":450114956044009477,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114956069961733} /tidb/cdc/default/default/changefeed/status/10c33d43-351b-4b6e-9ac7-b39ed4bd0736 {"checkpoint-ts":450114959071772674,"min-table-barrier-ts":450114959097724934,"admin-job-type":0} /tidb/cdc/default/default/task/position/a4eb1fdd-ccc1-4409-bde8-a7adeb529ab7/10c33d43-351b-4b6e-9ac7-b39ed4bd0736 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683371390909134 {"id":7374683371390909134,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":n+ grep -q 'failed to get info:' ull}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/10c33d43-351b-4b6e-9ac7-b39ed4bd0736 {UpstreamID:7374683371390909134 Namespace:default ID:10c33d43-351b-4b6e-9ac7-b39ed4bd0736 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:58:12.154676596 +0800 CST StartTs:450114956044009477 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00403acf0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114956069961733} {CheckpointTs:450114959071772674 MinTableBarrierTs:450114959071772674 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114959071772675, checkpointTs: 450114959071772674, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114959071772675, checkpointTs: 450114959071772674, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a4eb1fdd-ccc1-4409-bde8-a7adeb529ab7 {"id":"a4eb1fdd-ccc1-4409-bde8-a7adeb529ab7","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a77645e a4eb1fdd-ccc1-4409-bde8-a7adeb529ab7 /tidb/cdc/default/default/changefeed/info/10c33d43-351b-4b6e-9ac7-b39ed4bd0736 {"upstream-id":7374683371390909134,"namespace":"default","changefeed-id":"10c33d43-351b-4b6e-9ac7-b39ed4bd0736","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:58:12.154676596+08:00","start-ts":450114956044009477,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114956069961733} /tidb/cdc/default/default/changefeed/status/10c33d43-351b-4b6e-9ac7-b39ed4bd0736 {"checkpoint-ts":450114959071772674,"min-table-barrier-ts":450114959097724934,"admin-job-type":0} /tidb/cdc/default/default/task/position/a4eb1fdd-ccc1-4409-bde8-a7adeb529ab7/10c33d43-351b-4b6e-9ac7-b39ed4bd0736 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683371390909134 {"id":7374683371390909134,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check diff failed 5-th time, retry later owner pid: 3869 owner id 12ce2e67-c501-43d7-a518-4cd74ef17420 [Thu May 30 14:58:25 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/owner-run-with-error=1*return(true);github.com/pingcap/tiflow/cdc/capture/capture-resign-failed=1*return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.39753977.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_owner_retryable_error.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_owner_retryable_error.server2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 33-th check, retry later check diff successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:27 GMT < Content-Length: 681 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4765c873-e585-4f31-9cdb-18ceb0e25f8a {"id":"4765c873-e585-4f31-9cdb-18ceb0e25f8a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/22318fc849f52972 4765c873-e585-4f31-9cdb-18ceb0e25f8a /tidb/cdc/default/default/upstream/7374683233227573581 {"id":7374683233227573581,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4765c873-e585-4f31-9cdb-18ceb0e25f8a {"id":"4765c873-e585-4f31-9cdb-18ceb0e25f8a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/22318fc849f52972 4765c873-e585-4f31-9cdb-18ceb0e25f8a /tidb/cdc/default/default/upstream/7374683233227573581 {"id":7374683233227573581,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4765c873-e585-4f31-9cdb-18ceb0e25f8a {"id":"4765c873-e585-4f31-9cdb-18ceb0e25f8a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/22318fc849f52972 4765c873-e585-4f31-9cdb-18ceb0e25f8a /tidb/cdc/default/default/upstream/7374683233227573581 {"id":7374683233227573581,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6009.out cli changefeed create --start-ts=450114947678470148 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' -c changefeed-error-2 Create changefeed successfully! ID: changefeed-error-2 Info: {"upstream_id":7374683233227573581,"namespace":"default","id":"changefeed-error-2","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T14:58:27.447493945+08:00","start_ts":450114947678470148,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114947678470148,"checkpoint_ts":450114947678470148,"checkpoint_time":"2024-05-30 14:57:40.126"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212bc8000006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-gszd6, pid:3830, start at 2024-05-30 14:58:25.932709563 +0800 CST m=+6.307799254 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:25.953 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:25.920 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:25.920 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212bc8000006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-gszd6, pid:3830, start at 2024-05-30 14:58:25.932709563 +0800 CST m=+6.307799254 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:25.953 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:25.920 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:25.920 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212bc9600002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-gszd6, pid:3890, start at 2024-05-30 14:58:26.009014624 +0800 CST m=+6.293166727 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:26.030 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:26.008 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:26.008 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table ddl_manager.finish_mark not exists for 34-th check, retry later check diff failed 1-th time, retry later + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-error-2 failed [CDC:ErrSnapshotLostByGC] + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error-2 + expected_state=failed + error_msg='[CDC:ErrSnapshotLostByGC]' + tls_dir='[CDC:ErrSnapshotLostByGC]' + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error-2 -s + info='{ "upstream_id": 7374683233227573581, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 0, "checkpoint_time": "0001-01-01 00:00:00.000", "error": { "time": "2024-05-30T14:58:27.456240391+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 450114947678470148 is earlier than or equal to GC safepoint at 9223372036854775807" } }' + echo '{ "upstream_id": 7374683233227573581, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 0, "checkpoint_time": "0001-01-01 00:00:00.000", "error": { "time": "2024-05-30T14:58:27.456240391+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 450114947678470148 is earlier than or equal to GC safepoint at 9223372036854775807" } }' { "upstream_id": 7374683233227573581, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 0, "checkpoint_time": "0001-01-01 00:00:00.000", "error": { "time": "2024-05-30T14:58:27.456240391+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 450114947678470148 is earlier than or equal to GC safepoint at 9223372036854775807" } } ++ jq -r .state ++ echo '{' '"upstream_id":' 7374683233227573581, '"namespace":' '"default",' '"id":' '"changefeed-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 0, '"checkpoint_time":' '"0001-01-01' '00:00:00.000",' '"error":' '{' '"time":' '"2024-05-30T14:58:27.456240391+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSnapshotLostByGC",' '"message":' '"[CDC:ErrSnapshotLostByGC]fail' to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 450114947678470148 is earlier than or equal to GC safepoint at '9223372036854775807"' '}' '}' + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:28 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037ad290 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114960299393025 MinTableBarrierTs:450114960299393025 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/12ce2e67-c501-43d7-a518-4cd74ef17420 {"id":"12ce2e67-c501-43d7-a518-4cd74ef17420","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/cad0beb5-03ca-460e-8695-835db3878ac8 {"id":"cad0beb5-03ca-460e-8695-835db3878ac8","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525bef 12ce2e67-c501-43d7-a518-4cd74ef17420 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525c39 cad0beb5-03ca-460e-8695-835db3878ac8 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114960299393025,"min-table-barrier-ts":450114960299393025,"admin-job-type":0} /tidb/cdc/default/default/task/position/12ce2e67-c501-43d7-a518-4cd74ef17420/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/cad0beb5-03ca-460e-8695-835db3878ac8/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037ad290 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114960299393025 MinTableBarrierTs:450114960299393025 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/12ce2e67-c501-43d7-a518-4cd74ef17420 {"id":"12ce2e67-c501-43d7-a518-4cd74ef17420","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/cad0beb5-03ca-460e-8695-835db3878ac8 {"id":"cad0beb5-03ca-460e-8695-835db3878ac8","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525bef 12ce2e67-c501-43d7-a518-4cd74ef17420 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525c39 cad0beb5-03ca-460e-8695-835db3878ac8 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114960299393025,"min-table-barrier-ts":450114960299393025,"admin-job-type":0} /tidb/cdc/default/default/task/position/12ce2e67-c501-43d7-a518-4cd74ef17420/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/cad0beb5-03ca-460e-8695-835db3878ac8/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"",+ grep -q 'failed to get info:' "cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037ad290 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114960299393025 MinTableBarrierTs:450114960299393025 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/12ce2e67-c501-43d7-a518-4cd74ef17420 {"id":"12ce2e67-c501-43d7-a518-4cd74ef17420","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/cad0beb5-03ca-460e-8695-835db3878ac8 {"id":"cad0beb5-03ca-460e-8695-835db3878ac8","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525bef 12ce2e67-c501-43d7-a518-4cd74ef17420 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525c39 cad0beb5-03ca-460e-8695-835db3878ac8 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114960299393025,"min-table-barrier-ts":450114960299393025,"admin-job-type":0} /tidb/cdc/default/default/task/position/12ce2e67-c501-43d7-a518-4cd74ef17420/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/cad0beb5-03ca-460e-8695-835db3878ac8/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list 2>&1 | grep -v "12ce2e67-c501-43d7-a518-4cd74ef17420" | grep id + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ jq -r .error.message ++ echo '{' '"upstream_id":' 7374683233227573581, '"namespace":' '"default",' '"id":' '"changefeed-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 0, '"checkpoint_time":' '"0001-01-01' '00:00:00.000",' '"error":' '{' '"time":' '"2024-05-30T14:58:27.456240391+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSnapshotLostByGC",' '"message":' '"[CDC:ErrSnapshotLostByGC]fail' to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 450114947678470148 is earlier than or equal to GC safepoint at '9223372036854775807"' '}' '}' + message='[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 450114947678470148 is earlier than or equal to GC safepoint at 9223372036854775807' + [[ ! [CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 450114947678470148 is earlier than or equal to GC safepoint at 9223372036854775807 =~ \[CDC:ErrSnapshotLostByGC] ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6090.out cli changefeed remove -c changefeed-error-2 Changefeed remove successfully. ID: changefeed-error-2 CheckpointTs: 0 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... "cluster-id": "default" "id": "cad0beb5-03ca-460e-8695-835db3878ac8", "cluster-id": "default" run task successfully table ddl_manager.finish_mark not exists for 35-th check, retry later check diff failed 2-th time, retry later + set +x check_redo_resolved_ts 10c33d43-351b-4b6e-9ac7-b39ed4bd0736 450114960867459076 s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/ /tmp/tidb_cdc_test/consistent_replicate_gbk/cdc_data/redo/10c33d43-351b-4b6e-9ac7-b39ed4bd0736/meta global resolved ts 450114959386345505 not forward to 450114960867459076 run task failed 1-th time, retry later capture_id: cad0beb5-03ca-460e-8695-835db3878ac8 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 35 100 35 0 0 28524 0 --:--:-- --:--:-- --:--:-- 35000 { "status": true, "message": "" }ETCDCTL_API=3 etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix | grep '12ce2e67-c501-43d7-a518-4cd74ef17420' 12ce2e67-c501-43d7-a518-4cd74ef17420 run task successfully ps -C cdc.test -o pid= | awk '{print $1}' | wc -l | grep 2 2 run task successfully test_owner_retryable_error pass wait process cdc.test exit for 1-th time... wait process cdc.test exit for 1-th time... [Thu May 30 14:58:31 CST 2024] <<<<<< START cdc server in batch_add_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + GO_FAILPOINTS= + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.51595161.out server --log-file /tmp/tidb_cdc_test/batch_add_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/batch_add_table/cdc_data --cluster-id default + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 2-th time... check_redo_resolved_ts 10c33d43-351b-4b6e-9ac7-b39ed4bd0736 450114960867459076 s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/ /tmp/tidb_cdc_test/consistent_replicate_gbk/cdc_data/redo/10c33d43-351b-4b6e-9ac7-b39ed4bd0736/meta wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 14:58:32 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/redo/ChangefeedNewRedoManagerError=2*return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.61306132.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 run task successfully table ddl_manager.finish_mark not exists for 36-th check, retry later check diff failed 3-th time, retry later wait process cdc.test exit for 3-th time... wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit run test case test_gap_between_watch_capture [Thu May 30 14:58:33 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/sleep-in-owner-tick=1*sleep(6000)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.41204122.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_gap_between_watch_capture.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_gap_between_watch_capture.server1 --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit \033[0;36m<<< Run all test success >>>\033[0m =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/partition_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2024/05/30 14:58:34.259 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2024/05/30 14:58:34.260 +08:00] [INFO] [apply.go:78] ["set memory limit"] [memoryLimit=10737418240] [2024/05/30 14:58:34.260 +08:00] [INFO] [apply.go:94] ["Start http pprof server"] [addr=:6060] [2024/05/30 14:58:34.261 +08:00] [INFO] [s3.go:406] ["succeed to get bucket region from s3"] ["bucket region"=us-east-1] [2024/05/30 14:58:34.264 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2024/05/30 14:58:34.264 +08:00] [INFO] [redo.go:167] ["apply redo log starts"] [checkpointTs=450114959386345505] [resolvedTs=450114961168924675] [2024/05/30 14:58:34.265 +08:00] [INFO] [s3.go:406] ["succeed to get bucket region from s3"] ["bucket region"=us-east-1] [2024/05/30 14:58:34.265 +08:00] [INFO] [tz.go:36] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/05/30 14:58:34.265 +08:00] [WARN] [config.go:394] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/05/30 14:58:34.265 +08:00] [ERROR] [config.go:209] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2024/05/30 14:58:34.265 +08:00] [INFO] [s3.go:406] ["succeed to get bucket region from s3"] ["bucket region"=us-east-1] [2024/05/30 14:58:34.268 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=4.138491ms] [2024/05/30 14:58:34.268 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=4.277464ms] [2024/05/30 14:58:34.270 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22IGNORE_SPACE%2CNO_AUTO_VALUE_ON_ZERO%2CNO_ENGINE_SUBSTITUTION%2CALLOW_INVALID_DATES%2CONLY_FULL_GROUP_BY%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/05/30 14:58:34.273 +08:00] [INFO] [mysql.go:198] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2024/05/30 14:58:34.274 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:58:34.274 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:58:34.274 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:58:34.274 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:58:34.274 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:58:34.274 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:58:34.274 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:58:34.274 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:58:34.274 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:58:34.274 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:58:34.274 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:58:34.274 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:58:34.274 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:58:34.274 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:58:34.274 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:58:34.274 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:58:34.274 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2024/05/30 14:58:34.274 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2024/05/30 14:58:34.274 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2024/05/30 14:58:34.274 +08:00] [INFO] [tz.go:36] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/05/30 14:58:34.274 +08:00] [WARN] [config.go:394] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/05/30 14:58:34.274 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2024/05/30 14:58:34.274 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2024/05/30 14:58:34.274 +08:00] [ERROR] [config.go:209] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2024/05/30 14:58:34.274 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2024/05/30 14:58:34.274 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2024/05/30 14:58:34.274 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2024/05/30 14:58:34.274 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2024/05/30 14:58:34.274 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2024/05/30 14:58:34.274 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2024/05/30 14:58:34.274 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2024/05/30 14:58:34.274 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2024/05/30 14:58:34.274 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2024/05/30 14:58:34.274 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2024/05/30 14:58:34.274 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2024/05/30 14:58:34.279 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22ONLY_FULL_GROUP_BY%2CNO_AUTO_VALUE_ON_ZERO%2CNO_ENGINE_SUBSTITUTION%2CIGNORE_SPACE%2CALLOW_INVALID_DATES%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/05/30 14:58:34.280 +08:00] [INFO] [mysql_ddl_sink.go:99] ["MySQL DDL sink is created"] [namespace=] [changefeed=] [2024/05/30 14:58:34.281 +08:00] [WARN] [redo.go:288] ["apply DDL"] [ddl="{\"StartTs\":450114959386345499,\"CommitTs\":450114959386345505,\"Query\":\"create table consistent_replicate_gbk.GBKTABLE2 like consistent_replicate_gbk.GBKTABLE\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_gbk\",\"tbl-name\":\"GBKTABLE2\",\"tbl-id\":108,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":3,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [2024/05/30 14:58:34.281 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":450114959386345499,\"CommitTs\":450114959386345505,\"Query\":\"create table consistent_replicate_gbk.GBKTABLE2 like consistent_replicate_gbk.GBKTABLE\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_gbk\",\"tbl-name\":\"GBKTABLE2\",\"tbl-id\":108,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":3,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [namespace=] [changefeed=] [2024/05/30 14:58:34.282 +08:00] [INFO] [mysql_ddl_sink.go:119] ["Execute DDL failed, but error can be ignored"] [startTs=450114959386345499] [ddl="create table consistent_replicate_gbk.GBKTABLE2 like consistent_replicate_gbk.GBKTABLE"] [namespace=] [changefeed=] [error="Error 1050 (42S01): Table 'consistent_replicate_gbk.GBKTABLE2' already exists"] [2024/05/30 14:58:34.282 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:58:34.483 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}] [checkpointTs=450114961168924675] [2024/05/30 14:58:34.483 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}] [checkpointTs=450114961168924675] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:34 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/79986b6c-6e01-4f23-a3a4-78312117afb7 {"id":"79986b6c-6e01-4f23-a3a4-78312117afb7","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ac3740a 79986b6c-6e01-4f23-a3a4-78312117afb7 /tidb/cdc/default/default/upstream/7374683459601557707 {"id":7374683459601557707,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/79986b6c-6e01-4f23-a3a4-78312117afb7 {"id":"79986b6c-6e01-4f23-a3a4-78312117afb7","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ac3740a 79986b6c-6e01-4f23-a3a4-78312117afb7 /tidb/cdc/default/default/upstream/7374683459601557707 {"id":7374683459601557707,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/79986b6c-6e01-4f23-a3a4-78312117afb7 {"id":"79986b6c-6e01-4f23-a3a4-78312117afb7","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ac3740a 79986b6c-6e01-4f23-a3a4-78312117afb7 /tidb/cdc/default/default/upstream/7374683459601557707 {"id":7374683459601557707,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.cli.5203.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ [2024/05/30 14:58:34.584 +08:00] [INFO] [redo.go:223] ["apply redo log finishes"] [appliedLogCount=1] [appliedDDLCount=1] [currentCheckpoint=450114961168924675] [2024/05/30 14:58:34.584 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2024/05/30 14:58:34.584 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] [2024/05/30 14:58:34.584 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2024/05/30 14:58:34.584 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2024/05/30 14:58:34.584 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2024/05/30 14:58:34.584 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2024/05/30 14:58:34.584 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2024/05/30 14:58:34.584 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2024/05/30 14:58:34.584 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2024/05/30 14:58:34.584 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2024/05/30 14:58:34.584 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2024/05/30 14:58:34.584 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] [2024/05/30 14:58:34.584 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2024/05/30 14:58:34.584 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2024/05/30 14:58:34.584 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2024/05/30 14:58:34.584 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] Apply redo log successfully check diff successfully [Thu May 30 14:58:34 CST 2024] <<<<<< run test case consistent_replicate_gbk success! >>>>>> Exiting on signal: INTERRUPT table ddl_manager.finish_mark not exists for 37-th check, retry later check diff failed 4-th time, retry later Create changefeed successfully! ID: 16a61efd-5592-4bd3-9d8f-4aac33fb73e6 Info: {"upstream_id":7374683459601557707,"namespace":"default","id":"16a61efd-5592-4bd3-9d8f-4aac33fb73e6","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T14:58:35.028020222+08:00","start_ts":450114962029281284,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114962029281284,"checkpoint_ts":450114962029281284,"checkpoint_time":"2024-05-30 14:58:34.870"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:35 GMT < Content-Length: 681 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d8b3d99d-6f03-4d3b-9b31-4032ccdc0582 {"id":"d8b3d99d-6f03-4d3b-9b31-4032ccdc0582","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 6 /tidb/cdc/default/__cdc_meta__/owner/22318fc849f529b6 d8b3d99d-6f03-4d3b-9b31-4032ccdc0582 /tidb/cdc/default/default/upstream/7374683233227573581 {"id":7374683233227573581,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d8b3d99d-6f03-4d3b-9b31-4032ccdc0582 {"id":"d8b3d99d-6f03-4d3b-9b31-4032ccdc0582","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 6 /tidb/cdc/default/__cdc_meta__/owner/22318fc849f529b6 d8b3d99d-6f03-4d3b-9b31-4032ccdc0582 /tidb/cdc/default/default/upstream/7374683233227573581 {"id":7374683233227573581,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d8b3d99d-6f03-4d3b-9b31-4032ccdc0582 {"id":"d8b3d99d-6f03-4d3b-9b31-4032ccdc0582","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 6 /tidb/cdc/default/__cdc_meta__/owner/22318fc849f529b6 d8b3d99d-6f03-4d3b-9b31-4032ccdc0582 /tidb/cdc/default/default/upstream/7374683233227573581 {"id":7374683233227573581,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6175.out cli changefeed create --start-ts=0 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' -c changefeed-initialize-error Create changefeed successfully! ID: changefeed-initialize-error Info: {"upstream_id":7374683233227573581,"namespace":"default","id":"changefeed-initialize-error","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T14:58:36.290363851+08:00","start_ts":450114962371379201,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114962371379201,"checkpoint_ts":450114962371379201,"checkpoint_time":"2024-05-30 14:58:36.175"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x table ddl_manager.finish_mark not exists for 38-th check, retry later table batch_add_table.finish_mark not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > check diff failed 5-th time, retry later table testSync.usertable exists table testSync.simple1 exists table testSync.simple2 exists start tidb cluster in /tmp/tidb_cdc_test/partition_table Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7374683233227573581, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 450114962371379201, "checkpoint_time": "2024-05-30 14:58:36.175", "error": { "time": "2024-05-30T14:58:36.464007717+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' + echo '{ "upstream_id": 7374683233227573581, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 450114962371379201, "checkpoint_time": "2024-05-30 14:58:36.175", "error": { "time": "2024-05-30T14:58:36.464007717+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' { "upstream_id": 7374683233227573581, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 450114962371379201, "checkpoint_time": "2024-05-30 14:58:36.175", "error": { "time": "2024-05-30T14:58:36.464007717+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } } ++ echo '{' '"upstream_id":' 7374683233227573581, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"warning",' '"checkpoint_tso":' 450114962371379201, '"checkpoint_time":' '"2024-05-30' '14:58:36.175",' '"error":' '{' '"time":' '"2024-05-30T14:58:36.464007717+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .state + state=warning + [[ ! warning == \n\o\r\m\a\l ]] + echo 'changefeed state warning does not equal to normal' changefeed state warning does not equal to normal + exit 1 run task failed 1-th time, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc) 3309751296 bytes in 14.22 secs (232693900 bytes/sec) [Pipeline] { table ddl_manager.finish_mark not exists for 39-th check, retry later [Pipeline] cache table batch_add_table.finish_mark not exists for 2-th check, retry later check diff failed 6-th time, retry later < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:40 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7374683233227573581, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 450114962371379201, "checkpoint_time": "2024-05-30 14:58:36.175", "error": { "time": "2024-05-30T14:58:36.464007717+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' + echo '{ "upstream_id": 7374683233227573581, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 450114962371379201, "checkpoint_time": "2024-05-30 14:58:36.175", "error": { "time": "2024-05-30T14:58:36.464007717+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' { "upstream_id": 7374683233227573581, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 450114962371379201, "checkpoint_time": "2024-05-30 14:58:36.175", "error": { "time": "2024-05-30T14:58:36.464007717+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } } ++ echo '{' '"upstream_id":' 7374683233227573581, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"warning",' '"checkpoint_tso":' 450114962371379201, '"checkpoint_time":' '"2024-05-30' '14:58:36.175",' '"error":' '{' '"time":' '"2024-05-30T14:58:36.464007717+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .state + state=warning + [[ ! warning == \n\o\r\m\a\l ]] + echo 'changefeed state warning does not equal to normal' changefeed state warning does not equal to normal + exit 1 run task failed 2-th time, retry later + res=' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003b9d560 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114961085825025 MinTableBarrierTs:450114961085825025 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/91046e4c-c05a-4611-b70b-01c32e10fbd1 {"id":"91046e4c-c05a-4611-b70b-01c32e10fbd1","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525c85 91046e4c-c05a-4611-b70b-01c32e10fbd1 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114961085825025,"min-table-barrier-ts":450114961085825025,"admin-job-type":0} /tidb/cdc/default/default/task/position/91046e4c-c05a-4611-b70b-01c32e10fbd1/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003b9d560 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114961085825025 MinTableBarrierTs:450114961085825025 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/91046e4c-c05a-4611-b70b-01c32e10fbd1 {"id":"91046e4c-c05a-4611-b70b-01c32e10fbd1","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525c85 91046e4c-c05a-4611-b70b-01c32e10fbd1 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114961085825025,"min-table-barrier-ts":450114961085825025,"admin-job-type":0} /tidb/cdc/default/default/task/position/91046e4c-c05a-4611-b70b-01c32e10fbd1/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003b9d560 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114961085825025 MinTableBarrierTs:450114961085825025 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/91046e4c-c05a-4611-b70b-01c32e10fbd1 {"id":"91046e4c-c05a-4611-b70b-01c32e10fbd1","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525c85 91046e4c-c05a-4611-b70b-01c32e10fbd1 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114961085825025,"min-table-barrier-ts":450114961085825025,"admin-job-type":0} /tidb/cdc/default/default/task/position/91046e4c-c05a-4611-b70b-01c32e10fbd1/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' table ddl_manager.finish_mark not exists for 40-th check, retry later check diff successfully table batch_add_table.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... Starting Upstream TiDB... table ddl_manager.finish_mark not exists for 41-th check, retry later check diff failed 1-th time, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 14:58:42 CST 2024] <<<<<< run test case batch_add_table success! >>>>>> Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) "is-owner": true, run task successfully check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7374683233227573581, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 450114964141113347, "checkpoint_time": "2024-05-30 14:58:42.926", "error": null }' + echo '{ "upstream_id": 7374683233227573581, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 450114964141113347, "checkpoint_time": "2024-05-30 14:58:42.926", "error": null }' { "upstream_id": 7374683233227573581, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 450114964141113347, "checkpoint_time": "2024-05-30 14:58:42.926", "error": null } ++ echo '{' '"upstream_id":' 7374683233227573581, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 450114964141113347, '"checkpoint_time":' '"2024-05-30' '14:58:42.926",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374683233227573581, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 450114964141113347, '"checkpoint_time":' '"2024-05-30' '14:58:42.926",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6324.out cli changefeed pause -c changefeed-initialize-error =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/consistent_replicate_nfs/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 42-th check, retry later check diff failed 2-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) owner pid: 4125 owner id 91046e4c-c05a-4611-b70b-01c32e10fbd1 [Thu May 30 14:58:45 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/sleep-in-owner-tick=1*sleep(6000)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.42184220.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_gap_between_watch_capture.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_gap_between_watch_capture.server2 --cluster-id default --addr 127.0.0.1:8301 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error stopped changefeed new redo manager injected error + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=stopped + error_msg=changefeed + tls_dir=error + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7374683233227573581, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 450114964402995203, "checkpoint_time": "2024-05-30 14:58:43.925", "error": { "time": "2024-05-30T14:58:36.464007717+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' + echo '{ "upstream_id": 7374683233227573581, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 450114964402995203, "checkpoint_time": "2024-05-30 14:58:43.925", "error": { "time": "2024-05-30T14:58:36.464007717+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' { "upstream_id": 7374683233227573581, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 450114964402995203, "checkpoint_time": "2024-05-30 14:58:43.925", "error": { "time": "2024-05-30T14:58:36.464007717+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } } ++ echo '{' '"upstream_id":' 7374683233227573581, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"stopped",' '"checkpoint_tso":' 450114964402995203, '"checkpoint_time":' '"2024-05-30' '14:58:43.925",' '"error":' '{' '"time":' '"2024-05-30T14:58:36.464007717+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7374683233227573581, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"stopped",' '"checkpoint_tso":' 450114964402995203, '"checkpoint_time":' '"2024-05-30' '14:58:43.925",' '"error":' '{' '"time":' '"2024-05-30T14:58:36.464007717+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .error.message + message='changefeed new redo manager injected error' + [[ ! changefeed new redo manager injected error =~ changefeed ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6393.out cli changefeed resume -c changefeed-initialize-error table ddl_manager.finish_mark not exists for 43-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc) 3309751296 bytes in 6.07 secs (544906891 bytes/sec) [Pipeline] { [Pipeline] cache PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... check diff failed 3-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:48 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003a986c0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114965280391174 MinTableBarrierTs:450114965542273030 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7193a4b5-3209-410f-abfb-11565a0872ff {"id":"7193a4b5-3209-410f-abfb-11565a0872ff","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/91046e4c-c05a-4611-b70b-01c32e10fbd1 {"id":"91046e4c-c05a-4611-b70b-01c32e10fbd1","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525c85 91046e4c-c05a-4611-b70b-01c32e10fbd1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525ce4 7193a4b5-3209-410f-abfb-11565a0872ff /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114965280391174,"min-table-barrier-ts":450114965542273030,"admin-job-type":0} /tidb/cdc/default/default/task/position/7193a4b5-3209-410f-abfb-11565a0872ff/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/91046e4c-c05a-4611-b70b-01c32e10fbd1/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003a986c0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114965280391174 MinTableBarrierTs:450114965542273030 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7193a4b5-3209-410f-abfb-11565a0872ff {"id":"7193a4b5-3209-410f-abfb-11565a0872ff","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/91046e4c-c05a-4611-b70b-01c32e10fbd1 {"id":"91046e4c-c05a-4611-b70b-01c32e10fbd1","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525c85 91046e4c-c05a-4611-b70b-01c32e10fbd1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525ce4 7193a4b5-3209-410f-abfb-11565a0872ff /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114965280391174,"min-table-barrier-ts":450114965542273030,"admin-job-type":0} /tidb/cdc/default/default/task/position/7193a4b5-3209-410f-abfb-11565a0872ff/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/91046e4c-c05a-4611-b70b-01c32e10fbd1/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003a986c0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114965280391174 MinTableBarrierTs:450114965542273030 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7193a4b5-3209-410f-abfb-11565a0872ff {"id":"7193a4b5-3209-410f-abfb-11565a0872ff","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/91046e4c-c05a-4611-b70b-01c32e10fbd1 {"id":"91046e4c-c05a-4611-b70b-01c32e10fbd1","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525c85 91046e4c-c05a-4611-b70b-01c32e10fbd1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525ce4 7193a4b5-3209-410f-abfb-11565a0872ff /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114965280391174,"min-table-barrier-ts":450114965542273030,"admin-job-type":0} /tidb/cdc/default/default/task/position/7193a4b5-3209-410f-abfb-11565a0872ff/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/91046e4c-c05a-4611-b70b-01c32e10fbd1/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list 2>&1 | grep -v "91046e4c-c05a-4611-b70b-01c32e10fbd1" | grep id table ddl_manager.finish_mark not exists for 44-th check, retry later check diff failed 4-th time, retry later + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7374683233227573581, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 450114964402995203, "checkpoint_time": "2024-05-30 14:58:43.925", "error": null }' + echo '{ "upstream_id": 7374683233227573581, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 450114964402995203, "checkpoint_time": "2024-05-30 14:58:43.925", "error": null }' { "upstream_id": 7374683233227573581, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 450114964402995203, "checkpoint_time": "2024-05-30 14:58:43.925", "error": null } ++ echo '{' '"upstream_id":' 7374683233227573581, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 450114964402995203, '"checkpoint_time":' '"2024-05-30' '14:58:43.925",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374683233227573581, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 450114964402995203, '"checkpoint_time":' '"2024-05-30' '14:58:43.925",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6463.out cli changefeed remove -c changefeed-initialize-error VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212d2a0c001d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-mzktr, pid:9027, start at 2024-05-30 14:58:48.621354169 +0800 CST m=+5.503149548 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:48.630 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:48.629 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:48.629 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212d2a0c001d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-mzktr, pid:9027, start at 2024-05-30 14:58:48.621354169 +0800 CST m=+5.503149548 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:48.630 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:48.629 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:48.629 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212d2bc00009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-mzktr, pid:9112, start at 2024-05-30 14:58:48.697756739 +0800 CST m=+5.526933740 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:48.707 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:48.688 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:48.688 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/partition_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/partition_table/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } "cluster-id": "default" "id": "7193a4b5-3209-410f-abfb-11565a0872ff", "cluster-id": "default" run task successfully Changefeed remove successfully. ID: changefeed-initialize-error CheckpointTs: 450114964402995203 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 45-th check, retry later check diff failed 5-th time, retry later + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.cli.10466.out cli tso query --pd=http://127.0.0.1:2379 + set +x capture_id: 7193a4b5-3209-410f-abfb-11565a0872ff wait process cdc.test exit for 1-th time... start tidb cluster in /tmp/tidb_cdc_test/consistent_replicate_nfs Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... wait process cdc.test exit for 2-th time... table ddl_manager.finish_mark not exists for 46-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/multi_rocks/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 14:58:53 CST 2024] <<<<<< run test case changefeed_error success! >>>>>> + set +x + tso='450114966461874177 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114966461874177 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 14:58:53 CST 2024] <<<<<< START cdc server in partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.1051310515.out server --log-file /tmp/tidb_cdc_test/partition_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/partition_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Verifying downstream PD is started... check diff successfully table ddl_manager.finish_mark not exists for 47-th check, retry later check diff failed 1-th time, retry later nonempty select id, val from test.availability1 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:56 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ccaee650-5039-47a1-b295-6b4fbe3a93ef {"id":"ccaee650-5039-47a1-b295-6b4fbe3a93ef","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b1d5ebe ccaee650-5039-47a1-b295-6b4fbe3a93ef /tidb/cdc/default/default/upstream/7374683562855778797 {"id":7374683562855778797,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ccaee650-5039-47a1-b295-6b4fbe3a93ef {"id":"ccaee650-5039-47a1-b295-6b4fbe3a93ef","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b1d5ebe ccaee650-5039-47a1-b295-6b4fbe3a93ef /tidb/cdc/default/default/upstream/7374683562855778797 {"id":7374683562855778797,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ccaee650-5039-47a1-b295-6b4fbe3a93ef {"id":"ccaee650-5039-47a1-b295-6b4fbe3a93ef","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b1d5ebe ccaee650-5039-47a1-b295-6b4fbe3a93ef /tidb/cdc/default/default/upstream/7374683562855778797 {"id":7374683562855778797,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.cli.10556.out cli changefeed create --start-ts=450114966461874177 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ table ddl_manager.finish_mark not exists for 48-th check, retry later Create changefeed successfully! ID: 292ae8b1-70be-45d5-88eb-8fc84a2eb8eb Info: {"upstream_id":7374683562855778797,"namespace":"default","id":"292ae8b1-70be-45d5-88eb-8fc84a2eb8eb","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T14:58:56.828332549+08:00","start_ts":450114966461874177,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114966461874177,"checkpoint_ts":450114966461874177,"checkpoint_time":"2024-05-30 14:58:51.779"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... nonempty select id, val from test.availability1 where id=1 and val=1 run task successfully nonempty select id, val from test.availability1 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/multi_rocks Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc) 3309751296 bytes in 7.86 secs (421351537 bytes/sec) [Pipeline] { [Pipeline] cache Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... check diff failed 2-th time, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x table ddl_manager.finish_mark not exists for 49-th check, retry later nonempty select id, val from test.availability1 where id=1 and val=22 run task successfully empty select id, val from test.availability1 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 3-th time, retry later empty select id, val from test.availability1 where id=1 run task successfully nonempty select id, val from test.availability2 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later table ddl_manager.finish_mark not exists for 50-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Debugger for raftstore-v2 is used Debugger for raftstore-v2 is used Debugger for raftstore-v2 is used Debugger for raftstore-v2 is used ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 4-th time, retry later table ddl_manager.finish_mark not exists for 51-th check, retry later nonempty select id, val from test.availability2 where id=1 and val=1 run task successfully nonempty select id, val from test.availability2 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212e0bf40010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3, pid:6905, start at 2024-05-30 14:59:03.049665526 +0800 CST m=+5.089259862 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:03.056 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:03.037 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:03.037 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212e0bf40010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3, pid:6905, start at 2024-05-30 14:59:03.049665526 +0800 CST m=+5.089259862 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:03.056 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:03.037 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:03.037 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212e0d6c0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3, pid:6983, start at 2024-05-30 14:59:03.158865396 +0800 CST m=+5.128046964 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:03.165 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:03.131 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:03.131 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_replicate_nfs/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_replicate_nfs/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/consistent_replicate_nfs/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/consistent_replicate_nfs/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_replicate_nfs/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 5-th time, retry later table ddl_manager.finish_mark not exists for 52-th check, retry later nonempty select id, val from test.availability2 where id=1 and val=22 run task successfully empty select id, val from test.availability2 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 1-th time, retry later check data successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 14:59:05 CST 2024] <<<<<< run test case ddl_puller_lag success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/ddl_sequence/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Thu May 30 14:59:06 CST 2024] <<<<<< START cdc server in consistent_replicate_nfs case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_nfs.82978299.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_nfs/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_nfs/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff successfully table ddl_manager.finish_mark not exists for 53-th check, retry later empty select id, val from test.availability2 where id=1 run task successfully nonempty select id, val from test.availability3 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/ddl_sequence Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 54-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:09 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/253827e5-f818-4cfe-b3bc-18e1bb73f206 {"id":"253827e5-f818-4cfe-b3bc-18e1bb73f206","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b5737ba 253827e5-f818-4cfe-b3bc-18e1bb73f206 /tidb/cdc/default/default/upstream/7374683615659736361 {"id":7374683615659736361,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/253827e5-f818-4cfe-b3bc-18e1bb73f206 {"id":"253827e5-f818-4cfe-b3bc-18e1bb73f206","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b5737ba 253827e5-f818-4cfe-b3bc-18e1bb73f206 /tidb/cdc/default/default/upstream/7374683615659736361 {"id":7374683615659736361,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/253827e5-f818-4cfe-b3bc-18e1bb73f206 {"id":"253827e5-f818-4cfe-b3bc-18e1bb73f206","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b5737ba 253827e5-f818-4cfe-b3bc-18e1bb73f206 /tidb/cdc/default/default/upstream/7374683615659736361 {"id":7374683615659736361,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ***************** properties ***************** "readallfields"="true" "scanproportion"="0" "mysql.host"="127.0.0.1" "mysql.user"="root" "operationcount"="0" "requestdistribution"="uniform" "readproportion"="0" "insertproportion"="0" "updateproportion"="0" "dotransactions"="false" "mysql.port"="4000" "workload"="core" "threadcount"="10" "mysql.db"="consistent_replicate_nfs" "recordcount"="50000" "fieldcount"="3" ********************************************** nonempty select id, val from test.availability3 where id=1 and val=1 run task successfully nonempty select id, val from test.availability3 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later check diff failed 2-th time, retry later table ddl_manager.finish_mark not exists for 55-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release nonempty select id, val from test.availability3 where id=1 and val=22 run task successfully empty select id, val from test.availability3 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc) 3309751296 bytes in 12.73 secs (260006343 bytes/sec) check diff failed 3-th time, retry later [Pipeline] { ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] sh table ddl_manager.finish_mark not exists for 56-th check, retry later [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G10 Run cases: default_value simple cdc_server_tips event_filter PROW_JOB_ID=c548ef96-eb30-4461-bb19-187e41d862e3 JENKINS_NODE_COOKIE=c5eeecef-341e-4644-a4a8-52367a661b59 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/ GOLANG_VERSION=1.21.10 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l2n7g HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1796069089731416066","prowjobid":"c548ef96-eb30-4461-bb19-187e41d862e3","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#590 TEST_GROUP=G10 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089731416066 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-qwk3v GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-qwk3v ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l2n7g GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l2n7g GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=590 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/default_value/run.sh using Sink-Type: mysql... <<================= [Pipeline] sh Starting Upstream TiDB... empty select id, val from test.availability3 where id=1 run task successfully test_gap_between_watch_capture pass [Pipeline] sh Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G17 Run cases: clustered_index processor_resolved_ts_fallback PROW_JOB_ID=c548ef96-eb30-4461-bb19-187e41d862e3 JENKINS_NODE_COOKIE=16e0b615-3bbf-478a-b843-b68c98c21613 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/ GOLANG_VERSION=1.21.10 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4vpcx HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1796069089731416066","prowjobid":"c548ef96-eb30-4461-bb19-187e41d862e3","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#590 TEST_GROUP=G17 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089731416066 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-bfvrw GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4vpcx ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-bfvrw GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4vpcx GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=590 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/clustered_index/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh wait process cdc.test exit for 1-th time... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G13 Run cases: tiflash region_merge common_1 PROW_JOB_ID=c548ef96-eb30-4461-bb19-187e41d862e3 JENKINS_NODE_COOKIE=4d330f14-ad62-4fac-98a0-442445b96c4c BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/ GOLANG_VERSION=1.21.10 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-9tzt0 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1796069089731416066","prowjobid":"c548ef96-eb30-4461-bb19-187e41d862e3","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#590 TEST_GROUP=G13 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089731416066 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-sdwtj GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-sdwtj ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-9tzt0 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-9tzt0 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=590 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/tiflash/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G14 Run cases: big_txn changefeed_finish force_replicate_table PROW_JOB_ID=c548ef96-eb30-4461-bb19-187e41d862e3 JENKINS_NODE_COOKIE=83d1ad70-fa0e-4096-83cd-18eb8d713201 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/ GOLANG_VERSION=1.21.10 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-nk3lw HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1796069089731416066","prowjobid":"c548ef96-eb30-4461-bb19-187e41d862e3","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#590 TEST_GROUP=G14 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089731416066 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-xzf88 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-xzf88 ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-nk3lw GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-nk3lw GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=590 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/big_txn/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] sh wait process cdc.test exit for 2-th time... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G16 table ddl_manager.finish_mark not exists for 57-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/ddl_only_block_related_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G11 Run cases: resolve_lock move_table autorandom generate_column PROW_JOB_ID=c548ef96-eb30-4461-bb19-187e41d862e3 JENKINS_NODE_COOKIE=ffe003af-25e3-44d9-a319-37117d2bc40a BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/ GOLANG_VERSION=1.21.10 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t2zx2 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1796069089731416066","prowjobid":"c548ef96-eb30-4461-bb19-187e41d862e3","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#590 TEST_GROUP=G11 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089731416066 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-55jnd GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-55jnd ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t2zx2 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t2zx2 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=590 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/resolve_lock/run.sh using Sink-Type: mysql... <<================= [Pipeline] sh Run cases: owner_resign processor_etcd_worker_delay sink_hang PROW_JOB_ID=c548ef96-eb30-4461-bb19-187e41d862e3 JENKINS_NODE_COOKIE=a41bd45b-c83c-441e-9376-2e755caba651 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/ GOLANG_VERSION=1.21.10 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-p8f0f HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1796069089731416066","prowjobid":"c548ef96-eb30-4461-bb19-187e41d862e3","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#590 TEST_GROUP=G16 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089731416066 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-kjrs1 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-kjrs1 ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-p8f0f GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-p8f0f GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=590 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/owner_resign/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G20 Run cases: tidb_mysql_test ddl_reentrant multi_cdc_cluster PROW_JOB_ID=c548ef96-eb30-4461-bb19-187e41d862e3 JENKINS_NODE_COOKIE=3e62270f-c540-4af1-a1cb-d298fd1e566a BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/ GOLANG_VERSION=1.21.10 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-bcfm9 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1796069089731416066","prowjobid":"c548ef96-eb30-4461-bb19-187e41d862e3","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#590 TEST_GROUP=G20 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089731416066 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-gqzgb GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-bcfm9 ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-gqzgb GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-bcfm9 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=590 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/tidb_mysql_test/run.sh using Sink-Type: mysql... <<================= wait process cdc.test exit for 3-th time... [Pipeline] sh check diff failed 4-th time, retry later + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G21 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Run cases: bank kill_owner_with_ddl owner_remove_table_error PROW_JOB_ID=c548ef96-eb30-4461-bb19-187e41d862e3 JENKINS_NODE_COOKIE=4bd321b9-37dd-4dd0-afa4-bb09e8c665c2 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/ GOLANG_VERSION=1.21.10 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-0vnff HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1796069089731416066","prowjobid":"c548ef96-eb30-4461-bb19-187e41d862e3","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#590 TEST_GROUP=G21 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089731416066 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-961q0 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-961q0 ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-0vnff GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-0vnff GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=590 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/bank/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit run test case delete_owner_key [Thu May 30 14:59:16 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.46224624.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_gap_between_watch_capture.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_gap_between_watch_capture.server1 --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G19 Run cases: changefeed_fast_fail batch_update_to_no_batch changefeed_resume_with_checkpoint_ts PROW_JOB_ID=c548ef96-eb30-4461-bb19-187e41d862e3 JENKINS_NODE_COOKIE=0ab4f0e6-2f0c-4205-aa51-bc9dd5ef67a3 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/ GOLANG_VERSION=1.21.10 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-cvg08 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1796069089731416066","prowjobid":"c548ef96-eb30-4461-bb19-187e41d862e3","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#590 TEST_GROUP=G19 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089731416066 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/590/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-v14kh GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-cvg08 ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_590-v14kh GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-cvg08 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=590 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/changefeed_fast_fail/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 58-th check, retry later check diff successfully The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/ddl_only_block_related_table Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/owner_resign Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... The 1 times to try to start tidb cluster... table ddl_manager.finish_mark not exists for 59-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/bank Starting Upstream PD... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:19 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0041bc6c0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114972095873032 MinTableBarrierTs:450114973419962369 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114972095873032, checkpointTs: 450114972095873032, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114972095873032, checkpointTs: 450114972095873032, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114972095873032, checkpointTs: 450114972095873032, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d79dc4d6-c180-4aac-9406-0e938efcd048 {"id":"d79dc4d6-c180-4aac-9406-0e938efcd048","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525dd1 d79dc4d6-c180-4aac-9406-0e938efcd048 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114972095873032,"min-table-barrier-ts":450114973419962369,"admin-job-type":0} /tidb/cdc/default/default/task/position/d79dc4d6-c180-4aac-9406-0e938efcd048/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0041bc6c0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114972095873032 MinTableBarrierTs:450114973419962369 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114972095873032, checkpointTs: 450114972095873032, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114972095873032, checkpointTs: 450114972095873032, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114972095873032, checkpointTs: 450114972095873032, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d79dc4d6-c180-4aac-9406-0e938efcd048 {"id":"d79dc4d6-c180-4aac-9406-0e938efcd048","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525dd1 d79dc4d6-c180-4aac-9406-0e938efcd048 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114972095873032,"min-table-barrier-ts":450114973419962369,"admin-job-type":0} /tidb/cdc/default/default/task/position/d79dc4d6-c180-4aac-9406-0e938efcd048/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0041bc6c0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114972095873032 MinTableBarrierTs:450114973419962369 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114972095873032, checkpointTs: 450114972095873032, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114972095873032, checkpointTs: 450114972095873032, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114972095873032, checkpointTs: 450114972095873032, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d79dc4d6-c180-4aac-9406-0e938efcd048 {"id":"d79dc4d6-c180-4aac-9406-0e938efcd048","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525dd1 d79dc4d6-c180-4aac-9406-0e938efcd048 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114972095873032,"min-table-barrier-ts":450114973419962369,"admin-job-type":0} /tidb/cdc/default/default/task/position/d79dc4d6-c180-4aac-9406-0e938efcd048/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' check diff failed 1-th time, retry later Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... table partition_table.t exists table partition_table.t1 exists table partition_table.t2 not exists for 1-th check, retry later table partition_table.t2 exists table partition_table.finish_mark not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/default_value Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212f09ac0017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t5729, pid:7450, start at 2024-05-30 14:59:19.307187579 +0800 CST m=+5.187667133 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:19.314 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:19.275 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:19.275 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212f09ac0017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t5729, pid:7450, start at 2024-05-30 14:59:19.307187579 +0800 CST m=+5.187667133 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:19.314 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:19.275 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:19.275 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212f0b24000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t5729, pid:7533, start at 2024-05-30 14:59:19.384460072 +0800 CST m=+5.214711572 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:19.391 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:19.369 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:19.369 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_sequence/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_sequence/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } The 1 times to try to start tidb cluster... table partition_table.finish_mark not exists for 2-th check, retry later table ddl_manager.finish_mark not exists for 60-th check, retry later "is-owner": true, run task successfully Verifying downstream PD is started... table test.finish_mark not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later Verifying downstream PD is started... INSERT - Takes(s): 10.0, Count: 27555, OPS: 2756.8, Avg(us): 3587, Min(us): 681, Max(us): 92569, 95th(us): 3000, 99th(us): 84000 start tidb cluster in /tmp/tidb_cdc_test/tidb_mysql_test Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... table test.finish_mark not exists for 2-th check, retry later owner pid: 4627 owner id d79dc4d6-c180-4aac-9406-0e938efcd048 default owner key /tidb/cdc/default/__cdc_meta__/owner/22318fc849525dd1 [Thu May 30 14:59:22 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.47304732.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_gap_between_watch_capture.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_gap_between_watch_capture.server2 --cluster-id default --addr 127.0.0.1:8301 + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.cli.8797.out cli tso query --pd=http://127.0.0.1:2379 Verifying downstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/clustered_index Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table partition_table.finish_mark not exists for 3-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release start tidb cluster in /tmp/tidb_cdc_test/tiflash Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/big_txn Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 3-th time, retry later table ddl_manager.finish_mark not exists for 61-th check, retry later Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Verifying downstream PD is started... table test.finish_mark not exists for 3-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + tso='450114974574182401 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114974574182401 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 14:59:24 CST 2024] <<<<<< START cdc server in ddl_sequence case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.88238825.out server --log-file /tmp/tidb_cdc_test/ddl_sequence/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_sequence/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 start tidb cluster in /tmp/tidb_cdc_test/changefeed_fast_fail Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... table partition_table.finish_mark not exists for 4-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 62-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:25 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003999cb0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114974992826372 MinTableBarrierTs:450114974992826372 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/11682883-0243-494f-b6c4-7ebac8a76a83 {"id":"11682883-0243-494f-b6c4-7ebac8a76a83","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/d79dc4d6-c180-4aac-9406-0e938efcd048 {"id":"d79dc4d6-c180-4aac-9406-0e938efcd048","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525dd1 d79dc4d6-c180-4aac-9406-0e938efcd048 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525e09 11682883-0243-494f-b6c4-7ebac8a76a83 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114974992826372,"min-table-barrier-ts":450114974992826372,"admin-job-type":0} /tidb/cdc/default/default/task/position/11682883-0243-494f-b6c4-7ebac8a76a83/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/d79dc4d6-c180-4aac-9406-0e938efcd048/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003999cb0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114974992826372 MinTableBarrierTs:450114974992826372 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/11682883-0243-494f-b6c4-7ebac8a76a83 {"id":"11682883-0243-494f-b6c4-7ebac8a76a83","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/d79dc4d6-c180-4aac-9406-0e938efcd048 {"id":"d79dc4d6-c180-4aac-9406-0e938efcd048","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525dd1 d79dc4d6-c180-4aac-9406-0e938efcd048 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525e09 11682883-0243-494f-b6c4-7ebac8a76a83 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114974992826372,"min-table-barrier-ts":450114974992826372,"admin-job-type":0} /tidb/cdc/default/default/task/position/11682883-0243-494f-b6c4-7ebac8a76a83/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/d79dc4d6-c180-4aac-9406-0e938efcd048/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"",+ grep -q 'failed to get info:' "cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003999cb0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114974992826372 MinTableBarrierTs:450114974992826372 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/11682883-0243-494f-b6c4-7ebac8a76a83 {"id":"11682883-0243-494f-b6c4-7ebac8a76a83","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/d79dc4d6-c180-4aac-9406-0e938efcd048 {"id":"d79dc4d6-c180-4aac-9406-0e938efcd048","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525dd1 d79dc4d6-c180-4aac-9406-0e938efcd048 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525e09 11682883-0243-494f-b6c4-7ebac8a76a83 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114974992826372,"min-table-barrier-ts":450114974992826372,"admin-job-type":0} /tidb/cdc/default/default/task/position/11682883-0243-494f-b6c4-7ebac8a76a83/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/d79dc4d6-c180-4aac-9406-0e938efcd048/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep -v "d79dc4d6-c180-4aac-9406-0e938efcd048 default" | grep id Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release check diff failed 4-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/resolve_lock Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table test.finish_mark not exists for 4-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) "id": "11682883-0243-494f-b6c4-7ebac8a76a83", run task successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.finish_mark not exists for 5-th check, retry later Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 63-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:27 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/379554ee-54ad-4a87-8603-a6c502925560 {"id":"379554ee-54ad-4a87-8603-a6c502925560","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b9672c1 379554ee-54ad-4a87-8603-a6c502925560 /tidb/cdc/default/default/upstream/7374683694028720990 {"id":7374683694028720990,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/379554ee-54ad-4a87-8603-a6c502925560 {"id":"379554ee-54ad-4a87-8603-a6c502925560","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b9672c1 379554ee-54ad-4a87-8603-a6c502925560 /tidb/cdc/default/default/upstream/7374683694028720990 {"id":7374683694028720990,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/379554ee-54ad-4a87-8603-a6c502925560 {"id":"379554ee-54ad-4a87-8603-a6c502925560","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b9672c1 379554ee-54ad-4a87-8603-a6c502925560 /tidb/cdc/default/default/upstream/7374683694028720990 {"id":7374683694028720990,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.cli.8872.out cli changefeed create --start-ts=450114974574182401 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying downstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) capture_id: 11682883-0243-494f-b6c4-7ebac8a76a83 1 ETCDCTL_API=3 etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix | grep '11682883-0243-494f-b6c4-7ebac8a76a83' 11682883-0243-494f-b6c4-7ebac8a76a83 run task successfully curl -X GET http://127.0.0.1:8300/status | grep '"is_owner": false' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 176 100 176 0 0 27734 0 --:--:-- --:--:-- --:--:-- 29333 "is_owner": false run task successfully Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore table test.finish_mark not exists for 5-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.finish_mark not exists for 6-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 64-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Create changefeed successfully! ID: 19d8bfa7-aec3-4af6-8805-09900d1d41ea Info: {"upstream_id":7374683694028720990,"namespace":"default","id":"19d8bfa7-aec3-4af6-8805-09900d1d41ea","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T14:59:29.864877503+08:00","start_ts":450114974574182401,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114974574182401,"checkpoint_ts":450114974574182401,"checkpoint_time":"2024-05-30 14:59:22.725"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... INSERT - Takes(s): 20.0, Count: 36422, OPS: 1821.5, Avg(us): 5455, Min(us): 681, Max(us): 489977, 95th(us): 42000, 99th(us): 89000 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 6-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.finish_mark not exists for 7-th check, retry later table ddl_manager.finish_mark not exists for 65-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212fb7fc0018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-gszd6, pid:6287, start at 2024-05-30 14:59:30.469913316 +0800 CST m=+26.966208493 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:30.478 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:30.481 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:30.481 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212fb7fc0018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-gszd6, pid:6287, start at 2024-05-30 14:59:30.469913316 +0800 CST m=+26.966208493 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:30.478 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:30.481 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:30.481 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212ec578000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-gszd6, pid:6349, start at 2024-05-30 14:59:14.928507011 +0800 CST m=+11.327379255 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:14.942 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:14.910 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:14.910 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_rocks/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_rocks/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/multi_rocks/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/multi_rocks/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_rocks/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) nonempty select id, val from test.availability1 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212fc3d80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l2n7g, pid:1202, start at 2024-05-30 14:59:31.227461798 +0800 CST m=+5.596969334 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:31.237 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:31.240 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:31.240 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 7-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 1-th time, retry later table partition_table.finish_mark not exists for 8-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 14:59:33.340 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc00164c780/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212fd67c0002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-bcfm9, pid:1309, start at 2024-05-30 14:59:32.386059487 +0800 CST m=+5.366183475 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:32.394 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:32.383 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:32.383 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark exists check diff successfully nonempty select id, val from test.availability1 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 2-th time, retry later wait process cdc.test exit for 1-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 14:59:33.412 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0008601e0/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 14:59:33.412 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001a985a0/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 14:59:33.339 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0012d0000/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212fc88c0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-p8f0f, pid:1234, start at 2024-05-30 14:59:31.540988836 +0800 CST m=+6.716273178 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:31.559 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:31.541 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:31.541 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212fc88c0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-p8f0f, pid:1234, start at 2024-05-30 14:59:31.540988836 +0800 CST m=+6.716273178 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:31.559 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:31.541 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:31.541 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212fc3d80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l2n7g, pid:1202, start at 2024-05-30 14:59:31.227461798 +0800 CST m=+5.596969334 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:31.237 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:31.240 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:31.240 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212fc5400014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l2n7g, pid:1280, start at 2024-05-30 14:59:31.309759216 +0800 CST m=+5.618534808 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:31.319 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:31.280 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:31.280 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/default_value/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/default_value/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/default_value/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/default_value/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/default_value/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Thu May 30 14:59:33 CST 2024] <<<<<< START cdc server in multi_rocks case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_rocks.1925719259.out server --log-file /tmp/tidb_cdc_test/multi_rocks/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_rocks/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table test.finish_mark not exists for 8-th check, retry later wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later table partition_table.finish_mark not exists for 9-th check, retry later wait process cdc.test exit for 3-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212fd67c0002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-bcfm9, pid:1309, start at 2024-05-30 14:59:32.386059487 +0800 CST m=+5.366183475 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:32.394 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:32.383 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:32.383 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212fd7300015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-bcfm9, pid:1373, start at 2024-05-30 14:59:32.449342367 +0800 CST m=+5.352611850 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:32.456 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:32.428 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:32.428 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/tidb_mysql_test/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/tidb_mysql_test/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/tidb_mysql_test/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/tidb_mysql_test/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/tidb_mysql_test/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 14:59:35 CST 2024] <<<<<< run test case ddl_manager success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212fe704000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-p8f0f, pid:1270, start at 2024-05-30 14:59:33.470269072 +0800 CST m=+8.537076200 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:33.495 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:33.492 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:33.492 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/owner_resign/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/owner_resign/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.default_value.cli.2624.out cli tso query --pd=http://127.0.0.1:2379 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:36 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/67513ea8-c8df-4030-a78b-1cfa7d30d1dc {"id":"67513ea8-c8df-4030-a78b-1cfa7d30d1dc","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b6cdff3 67513ea8-c8df-4030-a78b-1cfa7d30d1dc /tidb/cdc/default/default/upstream/7374683648334018512 {"id":7374683648334018512,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/67513ea8-c8df-4030-a78b-1cfa7d30d1dc {"id":"67513ea8-c8df-4030-a78b-1cfa7d30d1dc","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b6cdff3 67513ea8-c8df-4030-a78b-1cfa7d30d1dc /tidb/cdc/default/default/upstream/7374683648334018512 {"id":7374683648334018512,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/67513ea8-c8df-4030-a78b-1cfa7d30d1dc {"id":"67513ea8-c8df-4030-a78b-1cfa7d30d1dc","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b6cdff3 67513ea8-c8df-4030-a78b-1cfa7d30d1dc /tidb/cdc/default/default/upstream/7374683648334018512 {"id":7374683648334018512,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_rocks.cli.19525.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ table test.finish_mark not exists for 9-th check, retry later Create changefeed successfully! ID: 1d553690-c14c-494c-9865-81fd65675121 Info: {"upstream_id":7374683648334018512,"namespace":"default","id":"1d553690-c14c-494c-9865-81fd65675121","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T14:59:37.019584602+08:00","start_ts":450114978285092870,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114978285092870,"checkpoint_ts":450114978285092870,"checkpoint_time":"2024-05-30 14:59:36.881"} PASS check diff failed 3-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.finish_mark not exists for 10-th check, retry later coverage: 2.4% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 14:59:37.491 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001e421e0/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tidb_mysql_test.cli.2601.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='450114978143272961 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114978143272961 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 14:59:37 CST 2024] <<<<<< START cdc server in default_value case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.default_value.26562658.out server --log-file /tmp/tidb_cdc_test/default_value/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/default_value/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_sequence.finish_mark not exists for 1-th check, retry later + set +x [Thu May 30 14:59:38 CST 2024] <<<<<< START cdc server in owner_resign case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.25142516.out server --log-file /tmp/tidb_cdc_test/owner_resign/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/owner_resign/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table test.finish_mark not exists for 10-th check, retry later nonempty select id, val from test.availability1 where id=1 and val=1 run task successfully nonempty select id, val from test.availability1 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 4-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.finish_mark not exists for 11-th check, retry later + set +x + tso='450114978508439553 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114978508439553 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 14:59:39 CST 2024] <<<<<< START cdc server in tidb_mysql_test case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tidb_mysql_test.26272629.out server --log-file /tmp/tidb_cdc_test/tidb_mysql_test/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/tidb_mysql_test/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21301bd00005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t2zx2, pid:1192, start at 2024-05-30 14:59:36.825234263 +0800 CST m=+5.647780858 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:36.835 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:36.820 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:36.820 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21301bd00005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t2zx2, pid:1192, start at 2024-05-30 14:59:36.825234263 +0800 CST m=+5.647780858 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:36.835 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:36.820 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:36.820 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213037e00010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t2zx2, pid:1259, start at 2024-05-30 14:59:38.636489577 +0800 CST m=+7.381812761 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:38.645 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:38.616 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:38.616 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ***************** properties ***************** "mysql.db"="multi_rocks" "threadcount"="2" "operationcount"="0" "table"="a1" "mysql.host"="127.0.0.1" "readproportion"="0" "updateproportion"="0" "insertproportion"="0" "readallfields"="true" "dotransactions"="false" "workload"="core" "requestdistribution"="uniform" "recordcount"="1000" "scanproportion"="0" "mysql.port"="4000" "mysql.user"="root" ********************************************** ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Logging trace to /tmp/tidb_cdc_test/resolve_lock/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/resolve_lock/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table ddl_sequence.finish_mark not exists for 2-th check, retry later INSERT - Takes(s): 30.0, Count: 48259, OPS: 1608.9, Avg(us): 5995, Min(us): 681, Max(us): 489977, 95th(us): 56000, 99th(us): 89000 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) nonempty select id, val from test.availability1 where id=1 and val=22 run task successfully empty select id, val from test.availability1 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 5-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.finish_mark exists check diff successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:40 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0b81a9fe-82be-405b-b9db-37e71136167f {"id":"0b81a9fe-82be-405b-b9db-37e71136167f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bc719c4 0b81a9fe-82be-405b-b9db-37e71136167f /tidb/cdc/default/default/upstream/7374683732547058637 {"id":7374683732547058637,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0b81a9fe-82be-405b-b9db-37e71136167f {"id":"0b81a9fe-82be-405b-b9db-37e71136167f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bc719c4 0b81a9fe-82be-405b-b9db-37e71136167f /tidb/cdc/default/default/upstream/7374683732547058637 {"id":7374683732547058637,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0b81a9fe-82be-405b-b9db-37e71136167f {"id":"0b81a9fe-82be-405b-b9db-37e71136167f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bc719c4 0b81a9fe-82be-405b-b9db-37e71136167f /tidb/cdc/default/default/upstream/7374683732547058637 {"id":7374683732547058637,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.default_value.cli.2704.out cli changefeed create --start-ts=450114978143272961 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:41 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7d19c98d-0898-4f32-9b97-d4f88393d553 {"id":"7d19c98d-0898-4f32-9b97-d4f88393d553","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bc01dc2 7d19c98d-0898-4f32-9b97-d4f88393d553 /tidb/cdc/default/default/upstream/7374683737977795660 {"id":7374683737977795660,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7d19c98d-0898-4f32-9b97-d4f88393d553 {"id":"7d19c98d-0898-4f32-9b97-d4f88393d553","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bc01dc2 7d19c98d-0898-4f32-9b97-d4f88393d553 /tidb/cdc/default/default/upstream/7374683737977795660 {"id":7374683737977795660,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7d19c98d-0898-4f32-9b97-d4f88393d553 {"id":"7d19c98d-0898-4f32-9b97-d4f88393d553","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bc01dc2 7d19c98d-0898-4f32-9b97-d4f88393d553 /tidb/cdc/default/default/upstream/7374683737977795660 {"id":7374683737977795660,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc cli capture list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' Create changefeed successfully! ID: a9678d8e-b673-4686-976e-5f65a414dd3e Info: {"upstream_id":7374683732547058637,"namespace":"default","id":"a9678d8e-b673-4686-976e-5f65a414dd3e","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T14:59:41.344291263+08:00","start_ts":450114978143272961,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114978143272961,"checkpoint_ts":450114978143272961,"checkpoint_time":"2024-05-30 14:59:36.340"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... Run finished, takes 31.531561083s INSERT - Takes(s): 31.5, Count: 50000, OPS: 1586.0, Avg(us): 6279, Min(us): 681, Max(us): 1209159, 95th(us): 60000, 99th(us): 89000 table consistent_replicate_nfs.usertable exists ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) 1 run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.cli.2579.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --server=127.0.0.1:8301 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.cli.2629.out cli tso query --pd=http://127.0.0.1:2379 wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Create changefeed successfully! ID: d26c0025-d034-423a-863e-4190f3b65151 Info: {"upstream_id":7374683737977795660,"namespace":"default","id":"d26c0025-d034-423a-863e-4190f3b65151","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T14:59:42.017876178+08:00","start_ts":450114979598696450,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114979598696450,"checkpoint_ts":450114979598696450,"checkpoint_time":"2024-05-30 14:59:41.892"} PASS ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) coverage: 2.4% of statements in github.com/pingcap/tiflow/... wait process cdc.test exit for 1-th time... table ddl_sequence.finish_mark not exists for 3-th check, retry later wait process cdc.test exit for 3-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:42 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/689d0750-f168-4c99-9b70-b288627ec3bb {"id":"689d0750-f168-4c99-9b70-b288627ec3bb","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bccc9c2 689d0750-f168-4c99-9b70-b288627ec3bb /tidb/cdc/default/default/upstream/7374683748620444205 {"id":7374683748620444205,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/689d0750-f168-4c99-9b70-b288627ec3bb {"id":"689d0750-f168-4c99-9b70-b288627ec3bb","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bccc9c2 689d0750-f168-4c99-9b70-b288627ec3bb /tidb/cdc/default/default/upstream/7374683748620444205 {"id":7374683748620444205,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/689d0750-f168-4c99-9b70-b288627ec3bb {"id":"689d0750-f168-4c99-9b70-b288627ec3bb","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bccc9c2 689d0750-f168-4c99-9b70-b288627ec3bb /tidb/cdc/default/default/upstream/7374683748620444205 {"id":7374683748620444205,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 142 0 0 100 142 0 1393 --:--:-- --:--:-- --:--:-- 1405 mysql_test start go: downloading github.com/maxshuang/mysql-tester v0.0.2 empty select id, val from test.availability1 where id=1 run task successfully nonempty select id, val from test.availability2 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/google/uuid v1.6.0 go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20240527072219-a97e6464c01d go: downloading github.com/BurntSushi/toml v1.3.2 go: downloading golang.org/x/time v0.5.0 go: downloading github.com/pingcap/failpoint v0.0.0-20240412033321-fd0796e60f86 go: downloading golang.org/x/sync v0.7.0 go: downloading google.golang.org/grpc v1.64.0 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20240527072219-a97e6464c01d go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 Run finished, takes 2.868776134s INSERT - Takes(s): 2.9, Count: 1000, OPS: 350.5, Avg(us): 3690, Min(us): 1052, Max(us): 1825875, 95th(us): 3000, 99th(us): 8000 table test.finish_mark not exists for 11-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 14:59:42 CST 2024] <<<<<< run test case partition_table success! >>>>>> go: downloading go.uber.org/atomic v1.11.0 go: downloading go.uber.org/multierr v1.11.0 wait process cdc.test exit for 3-th time... go: downloading golang.org/x/sys v0.20.0 go: downloading google.golang.org/protobuf v1.34.1 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240521202816-d264139d666e go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully [2024/05/30 14:59:43.492 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001e421e0/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + tso='450114979606036484 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114979606036484 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 14:59:43 CST 2024] <<<<<< START cdc server in resolve_lock case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.26642666.out server --log-file /tmp/tidb_cdc_test/resolve_lock/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/resolve_lock/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' 1 run task successfully table owner_resign.t1 not exists for 1-th check, retry later table test.finish_mark not exists for 12-th check, retry later table ddl_sequence.finish_mark not exists for 4-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ***************** properties ***************** "table"="a2" "readallfields"="true" "operationcount"="0" "recordcount"="1000" "mysql.db"="multi_rocks" "mysql.port"="4000" "readproportion"="0" "scanproportion"="0" "requestdistribution"="uniform" "mysql.host"="127.0.0.1" "workload"="core" "threadcount"="2" "insertproportion"="0" "dotransactions"="false" "mysql.user"="root" "updateproportion"="0" ********************************************** nonempty select id, val from test.availability2 where id=1 and val=1 run task successfully nonempty select id, val from test.availability2 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later ____________________________________ *************************** 1. row *************************** primary_ts: 450114913662402562 secondary_ts: 450114916321067023 *************************** 2. row *************************** primary_ts: 450114921526722560 secondary_ts: 450114921826091009 *************************** 3. row *************************** primary_ts: 450114929391042560 secondary_ts: 450114929690673153 *************************** 4. row *************************** primary_ts: 450114937255362560 secondary_ts: 450114937568100353 *************************** 5. row *************************** primary_ts: 450114945119682560 secondary_ts: 450114945432158209 *************************** 6. row *************************** primary_ts: 450114952984002560 secondary_ts: 450114953060810753 *************************** 7. row *************************** primary_ts: 450114960848322560 secondary_ts: 450114961187274753 *************************** 8. row *************************** primary_ts: 450114968712642560 secondary_ts: 450114968789188609 *************************** 9. row *************************** primary_ts: 450114976576962560 secondary_ts: 450114976679723009 skip invalid syncpoint primary_ts: 450114913662402562, first_ddl_ts: 450114917057167363 check diff successfully check diff successfully check diff successfully check diff successfully check diff successfully check diff successfully check diff successfully check diff successfully check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 14:59:39 CST 2024] <<<<<< run test case syncpoint success! >>>>>> go: downloading golang.org/x/text v0.15.0 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/pingcap/kvproto v0.0.0-20240522024016-df42997c2c57 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20240510070134-e7f530c64790 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/tikv/pd/client v0.0.0-20240210135946-3488a653ddd9 go: downloading github.com/golang/protobuf v1.5.4 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842 go: downloading github.com/pingcap/tipb v0.0.0-20240507090649-2bf6bb0cb996 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/prometheus/client_golang v1.19.1 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/shirou/gopsutil/v3 v3.24.4 go: downloading github.com/danjacques/gofslock v0.0.0-20220131014315-6e321f4509c8 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/gorilla/mux v1.8.0 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/google/btree v1.1.2 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/cockroachdb/pebble v0.0.0-20220415182917-06c9d3be25b3 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/pkg/errors v0.9.1 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/aws/aws-sdk-go v1.44.259 go: downloading github.com/stretchr/testify v1.9.0 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b check diff failed 1-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/multi_source/run.sh using Sink-Type: mysql... <<================= VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21307c6c0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-0vnff, pid:818, start at 2024-05-30 14:59:43.030781398 +0800 CST m=+18.187665401 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:43.047 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:43.053 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:43.053 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21307c6c0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-0vnff, pid:818, start at 2024-05-30 14:59:43.030781398 +0800 CST m=+18.187665401 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:43.047 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:43.053 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:43.053 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21308064000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-0vnff, pid:880, start at 2024-05-30 14:59:43.296622211 +0800 CST m=+18.353854959 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:43.397 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:43.407 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:43.407 +0800 All versions after safe point can be accessed. (DO NOT EDIT) go: downloading github.com/klauspost/compress v1.17.8 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.6 go: downloading cloud.google.com/go v0.112.2 go: downloading google.golang.org/api v0.170.0 go: downloading golang.org/x/oauth2 v0.18.0 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Run finished, takes 846.860886ms INSERT - Takes(s): 0.8, Count: 1000, OPS: 1212.0, Avg(us): 1651, Min(us): 950, Max(us): 21678, 95th(us): 3000, 99th(us): 5000 go: downloading github.com/pingcap/errors v0.11.5-0.20211009033009-93128226aaa3 go: downloading github.com/sirupsen/logrus v1.8.1 go: downloading github.com/go-sql-driver/mysql v1.6.0 go: downloading github.com/pingcap/tidb/parser v0.0.0-20211122065349-4d16aabf900b go: downloading go.uber.org/atomic v1.9.0 go: downloading golang.org/x/sys v0.0.0-20211117180635-dee7805ff2e1 go: downloading golang.org/x/text v0.3.7 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading go.uber.org/zap v1.19.1 go: downloading github.com/pingcap/log v0.0.0-20210906054005-afc726e70354 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.0.0 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20200410134404-eec4a21b6bb0 go: downloading go.uber.org/multierr v1.7.0 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading golang.org/x/net v0.25.0 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/iancoleman/strcase v0.2.0 go: downloading github.com/prometheus/common v0.53.0 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.0 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.1 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.1 go: downloading golang.org/x/crypto v0.23.0 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/prometheus/procfs v0.15.0 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/dgraph-io/ristretto v0.1.1 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 14:59:45.806 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0017661e0/127.0.0.1:2379] [attempt=0] [error="rpc error: code = NotFound desc = etcdserver: requested lease not found"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 14:59:45.808 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001e961e0/127.0.0.1:2479] [attempt=0] [error="rpc error: code = NotFound desc = etcdserver: requested lease not found"] ***************** properties ***************** "mysql.host"="127.0.0.1" "workload"="core" "dotransactions"="false" "mysql.db"="multi_rocks" "readallfields"="true" "mysql.port"="4000" "readproportion"="0" "threadcount"="2" "updateproportion"="0" "insertproportion"="0" "operationcount"="0" "requestdistribution"="uniform" "recordcount"="1000" "scanproportion"="0" "table"="a3" "mysql.user"="root" ********************************************** go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/fatih/color v1.16.0 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/mattn/go-runewidth v0.0.15 go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading github.com/sasha-s/go-deadlock v0.3.1 go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/pingcap/badger v1.5.1-0.20230103063557-828f39b09b6d go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 table ddl_sequence.finish_mark not exists for 5-th check, retry later go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/rivo/uniseg v0.4.7 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading github.com/petermattis/goid v0.0.0-20231207134359-e60b3f734c67 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.12.0 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/golang-jwt/jwt/v5 v5.2.0 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table owner_resign.t1 exists check diff failed 1-th time, retry later go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/klauspost/cpuid v1.3.1 table test.finish_mark not exists for 13-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:46 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f1fd0800-54b0-4d82-bb89-41a1d4d2e73a {"id":"f1fd0800-54b0-4d82-bb89-41a1d4d2e73a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bdaf2c3 f1fd0800-54b0-4d82-bb89-41a1d4d2e73a /tidb/cdc/default/default/upstream/7374683768182076119 {"id":7374683768182076119,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f1fd0800-54b0-4d82-bb89-41a1d4d2e73a {"id":"f1fd0800-54b0-4d82-bb89-41a1d4d2e73a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bdaf2c3 f1fd0800-54b0-4d82-bb89-41a1d4d2e73a /tidb/cdc/default/default/upstream/7374683768182076119 {"id":7374683768182076119,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f1fd0800-54b0-4d82-bb89-41a1d4d2e73a {"id":"f1fd0800-54b0-4d82-bb89-41a1d4d2e73a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bdaf2c3 f1fd0800-54b0-4d82-bb89-41a1d4d2e73a /tidb/cdc/default/default/upstream/7374683768182076119 {"id":7374683768182076119,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.cli.2711.out cli changefeed create --start-ts=450114979606036484 --sink-uri=mysql://normal:123456@127.0.0.1:3306/tidb-txn-mode=pessimistic go: downloading github.com/go-logr/logr v1.4.1 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading github.com/DataDog/zstd v1.5.5 nonempty select id, val from test.availability2 where id=1 and val=22 run task successfully empty select id, val from test.availability2 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later Create changefeed successfully! ID: c20e03f2-1179-4b47-8b16-f0ee36b4b3ab Info: {"upstream_id":7374683768182076119,"namespace":"default","id":"c20e03f2-1179-4b47-8b16-f0ee36b4b3ab","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/tidb-txn-mode=pessimistic","create_time":"2024-05-30T14:59:46.975775204+08:00","start_ts":450114979606036484,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114979606036484,"checkpoint_ts":450114979606036484,"checkpoint_time":"2024-05-30 14:59:41.920"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/bank/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/bank/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/bank/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/bank/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/bank/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Run finished, takes 979.089058ms INSERT - Takes(s): 1.0, Count: 1000, OPS: 1043.2, Avg(us): 1910, Min(us): 938, Max(us): 70467, 95th(us): 3000, 99th(us): 6000 ***************** properties ***************** "updateproportion"="0" "mysql.port"="4000" "recordcount"="1000" "workload"="core" "readproportion"="0" "scanproportion"="0" "requestdistribution"="uniform" "insertproportion"="0" "mysql.user"="root" "threadcount"="2" "dotransactions"="false" "table"="a4" "operationcount"="0" "mysql.host"="127.0.0.1" "readallfields"="true" "mysql.db"="multi_rocks" ********************************************** ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_sequence.finish_mark not exists for 6-th check, retry later check diff successfully [Thu May 30 14:59:48 CST 2024] <<<<<< START cdc server in owner_resign case >>>>>> + set +x go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/tikv/pd/client v0.0.0-20240210135946-3488a653ddd9 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20240510070134-e7f530c64790 go: downloading github.com/pingcap/kvproto v0.0.0-20240522024016-df42997c2c57 go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20240527072219-a97e6464c01d go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20240527072219-a97e6464c01d Run finished, takes 804.213579ms INSERT - Takes(s): 0.8, Count: 999, OPS: 1272.6, Avg(us): 1546, Min(us): 877, Max(us): 19088, 95th(us): 3000, 99th(us): 5000 + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerDontUpdateCheckpoint=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.27062708.out server --log-file /tmp/tidb_cdc_test/owner_resign/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/owner_resign/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table test.finish_mark not exists for 14-th check, retry later go: downloading github.com/BurntSushi/toml v1.3.2 go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/pingcap/failpoint v0.0.0-20240412033321-fd0796e60f86 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading go.uber.org/atomic v1.11.0 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading go.uber.org/multierr v1.11.0 ***************** properties ***************** "threadcount"="2" "operationcount"="0" "scanproportion"="0" "recordcount"="1000" "mysql.db"="multi_rocks" "insertproportion"="0" "mysql.user"="root" "table"="a5" "mysql.host"="127.0.0.1" "readallfields"="true" "requestdistribution"="uniform" "updateproportion"="0" "workload"="core" "readproportion"="0" "mysql.port"="4000" "dotransactions"="false" ********************************************** empty select id, val from test.availability2 where id=1 run task successfully nonempty select id, val from test.availability3 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2130a8d00001 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6w171, pid:5973, start at 2024-05-30 14:59:45.859543024 +0800 CST m=+21.648742741 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:46.289 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:46.344 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:46.344 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2130a8d00001 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6w171, pid:5973, start at 2024-05-30 14:59:45.859543024 +0800 CST m=+21.648742741 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:46.289 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:46.344 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:46.344 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21307a9c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6w171, pid:6039, start at 2024-05-30 14:59:42.917214164 +0800 CST m=+18.618347500 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:42.928 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:42.937 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:42.937 +0800 All versions after safe point can be accessed. (DO NOT EDIT) go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/google/uuid v1.6.0 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading github.com/google/btree v1.1.2 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading github.com/golang/protobuf v1.5.4 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/prometheus/client_golang v1.19.1 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading google.golang.org/grpc v1.64.0 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 go: downloading golang.org/x/sync v0.7.0 go: downloading google.golang.org/protobuf v1.34.1 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/prometheus/common v0.53.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/prometheus/procfs v0.15.0 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading golang.org/x/sys v0.20.0 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/hang_sink_suicide/run.sh using Sink-Type: mysql... <<================= [Thu May 30 14:59:48 CST 2024] <<<<<< run test case hang_sink_suicide success! >>>>>> go: downloading golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240521202816-d264139d666e go: downloading golang.org/x/net v0.25.0 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading golang.org/x/text v0.15.0 check diff failed 3-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_sequence.finish_mark exists ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... check diff successfully table test.finish_mark not exists for 15-th check, retry later Logging trace to /tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 1-th time... go: downloading github.com/jmespath/go-jmespath v0.4.0 go: downloading github.com/google/s2a-go v0.1.7 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading github.com/felixge/httpsnoop v1.0.4 nonempty select id, val from test.availability3 where id=1 and val=1 run task successfully nonempty select id, val from test.availability3 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later Run finished, takes 2.646534543s INSERT - Takes(s): 2.6, Count: 1000, OPS: 380.8, Avg(us): 3627, Min(us): 873, Max(us): 706816, 95th(us): 3000, 99th(us): 77000 cdc.test: no process found wait process cdc.test exit for 2-th time... process cdc.test already exit [Thu May 30 14:59:51 CST 2024] <<<<<< run test case ddl_sequence success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2130eb340013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4vpcx, pid:1195, start at 2024-05-30 14:59:50.135386501 +0800 CST m=+22.526970154 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:50.144 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:50.143 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:50.143 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2130eb340013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4vpcx, pid:1195, start at 2024-05-30 14:59:50.135386501 +0800 CST m=+22.526970154 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:50.144 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:50.143 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:50.143 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2130e8140014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4vpcx, pid:1273, start at 2024-05-30 14:59:49.929980933 +0800 CST m=+20.697077204 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:49.938 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:49.943 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:49.943 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/clustered_index/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/clustered_index/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/clustered_index/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/clustered_index/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/clustered_index/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2130e90c0015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-nk3lw, pid:1222, start at 2024-05-30 14:59:49.988330999 +0800 CST m=+20.735570334 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:49.997 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:50.004 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:50.004 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2130e90c0015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-nk3lw, pid:1222, start at 2024-05-30 14:59:49.988330999 +0800 CST m=+20.735570334 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:49.997 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:50.004 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:50.004 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2130ec080015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-nk3lw, pid:1275, start at 2024-05-30 14:59:50.192608183 +0800 CST m=+20.864421624 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:50.203 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:50.196 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:50.196 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/pingcap/tipb v0.0.0-20240507090649-2bf6bb0cb996 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/gorilla/mux v1.8.0 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/danjacques/gofslock v0.0.0-20220131014315-6e321f4509c8 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/aws/aws-sdk-go v1.44.259 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/cockroachdb/pebble v0.0.0-20220415182917-06c9d3be25b3 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/klauspost/compress v1.17.8 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/shirou/gopsutil/v3 v3.24.4 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.1 go: downloading github.com/stretchr/testify v1.9.0 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.1 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/xitongsys/parquet-go v1.6.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading golang.org/x/time v0.5.0 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.6 go: downloading golang.org/x/oauth2 v0.18.0 go: downloading google.golang.org/api v0.170.0 go: downloading github.com/sasha-s/go-deadlock v0.3.1 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/iancoleman/strcase v0.2.0 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0 go: downloading github.com/fatih/color v1.16.0 go: downloading github.com/vbauerster/mpb/v7 v7.5.3 go: downloading golang.org/x/term v0.20.0 go: downloading github.com/spf13/cobra v1.8.0 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/mattn/go-runewidth v0.0.15 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/petermattis/goid v0.0.0-20231207134359-e60b3f734c67 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.1 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.1 go: downloading golang.org/x/crypto v0.23.0 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/Masterminds/semver v1.5.0 check diff failed 4-th time, retry later TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/big_txn/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/big_txn/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/big_txn/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/big_txn/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/big_txn/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } go: downloading k8s.io/api v0.28.6 go: downloading github.com/robfig/cron v1.2.0 go: downloading cloud.google.com/go v0.112.2 go: downloading github.com/emirpasic/gods v1.18.1 go: downloading github.com/pingcap/badger v1.5.1-0.20230103063557-828f39b09b6d go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/golang-jwt/jwt/v5 v5.2.0 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/rogpeppe/go-internal v1.12.0 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/lestrrat-go/httpcc v1.0.1 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:51 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/d26c0025-d034-423a-863e-4190f3b65151 {UpstreamID:7374683737977795660 Namespace:default ID:d26c0025-d034-423a-863e-4190f3b65151 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:59:42.017876178 +0800 CST StartTs:450114979598696450 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003c321b0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114979624648709} {CheckpointTs:450114981944885262 MinTableBarrierTs:450114981944885262 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3ea026cf-ff8e-431c-931f-0039f514e2f6 {"id":"3ea026cf-ff8e-431c-931f-0039f514e2f6","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/7d19c98d-0898-4f32-9b97-d4f88393d553 {"id":"7d19c98d-0898-4f32-9b97-d4f88393d553","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bc01dc2 7d19c98d-0898-4f32-9b97-d4f88393d553 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bc01e53 3ea026cf-ff8e-431c-931f-0039f514e2f6 /tidb/cdc/default/default/changefeed/info/d26c0025-d034-423a-863e-4190f3b65151 {"upstream-id":7374683737977795660,"namespace":"default","changefeed-id":"d26c0025-d034-423a-863e-4190f3b65151","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:59:42.017876178+08:00","start-ts":450114979598696450,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114979624648709} /tidb/cdc/default/default/changefeed/status/d26c0025-d034-423a-863e-4190f3b65151 {"checkpoint-ts":450114981944885262,"min-table-barrier-ts":450114981944885262,"admin-job-type":0} /tidb/cdc/default/default/task/position/3ea026cf-ff8e-431c-931f-0039f514e2f6/d26c0025-d034-423a-863e-4190f3b65151 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/7d19c98d-0898-4f32-9b97-d4f88393d553/d26c0025-d034-423a-863e-4190f3b65151 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683737977795660 {"id":7374683737977795660,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/d26c0025-d034-423a-863e-4190f3b65151 {UpstreamID:7374683737977795660 Namespace:default ID:d26c0025-d034-423a-863e-4190f3b65151 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:59:42.017876178 +0800 CST StartTs:450114979598696450 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003c321b0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114979624648709} {CheckpointTs:450114981944885262 MinTableBarrierTs:450114981944885262 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3ea026cf-ff8e-431c-931f-0039f514e2f6 {"id":"3ea026cf-ff8e-431c-931f-0039f514e2f6","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/7d19c98d-0898-4f32-9b97-d4f88393d553 {"id":"7d19c98d-0898-4f32-9b97-d4f88393d553","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bc01dc2 7d19c98d-0898-4f32-9b97-d4f88393d553 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bc01e53 3ea026cf-ff8e-431c-931f-0039f514e2f6 /tidb/cdc/default/default/changefeed/info/d26c0025-d034-423a-863e-4190f3b65151 {"upstream-id":7374683737977795660,"namespace":"default","changefeed-id":"d26c0025-d034-423a-863e-4190f3b65151","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:59:42.017876178+08:00","start-ts":450114979598696450,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114979624648709} /tidb/cdc/default/default/changefeed/status/d26c0025-d034-423a-863e-4190f3b65151 {"checkpoint-ts":450114981944885262,"min-table-barrier-ts":450114981944885262,"admin-job-type":0} /tidb/cdc/default/default/task/position/3ea026cf-ff8e-431c-931f-0039f514e2f6/d26c0025-d034-423a-863e-4190f3b65151 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/7d19c98d-0898-4f32-9b97-d4f88393d553/d26c0025-d034-423a-863e-4190f3b65151 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683737977795660 {"id":7374683737977795660,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/d26c0025-d034-423a-863e-4190f3b65151 {UpstreamID:7374683737977795660 Namespace:default ID:d26c0025-d034-423a-863e-4190f3b65151 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:59:42.017876178 +0800 CST StartTs:450114979598696450 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003c321b0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114979624648709} {CheckpointTs:450114981944885262 MinTableBarrierTs:450114981944885262 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3ea026cf-ff8e-431c-931f-0039f514e2f6 {"id":"3ea026cf-ff8e-431c-931f-0039f514e2f6","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/7d19c98d-0898-4f32-9b97-d4f88393d553 {"id":"7d19c98d-0898-4f32-9b97-d4f88393d553","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bc01dc2 7d19c98d-0898-4f32-9b97-d4f88393d553 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bc01e53 3ea026cf-ff8e-431c-931f-0039f514e2f6 /tidb/cdc/default/default/changefeed/info/d26c0025-d034-423a-863e-4190f3b65151 {"upstream-id":7374683737977795660,"namespace":"default","changefeed-id":"d26c0025-d034-423a-863e-4190f3b65151","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:59:42.017876178+08:00","start-ts":450114979598696450,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114979624648709} /tidb/cdc/default/default/changefeed/status/d26c0025-d034-423a-863e-4190f3b65151 {"checkpoint-ts":450114981944885262,"min-table-barrier-ts":450114981944885262,"admin-job-type":0} /tidb/cdc/default/default/task/position/3ea026cf-ff8e-431c-931f-0039f514e2f6/d26c0025-d034-423a-863e-4190f3b65151 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/7d19c98d-0898-4f32-9b97-d4f88393d553/d26c0025-d034-423a-863e-4190f3b65151 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683737977795660 {"id":7374683737977795660,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^2$' 2 run task successfully % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 The 1 times to try to start tidb cluster... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2130ec24000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-cvg08, pid:848, start at 2024-05-30 14:59:50.163619181 +0800 CST m=+16.607966927 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:50.171 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:50.153 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:50.153 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2130ec24000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-cvg08, pid:848, start at 2024-05-30 14:59:50.163619181 +0800 CST m=+16.607966927 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:50.171 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:50.153 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:50.153 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2130f2300018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-cvg08, pid:935, start at 2024-05-30 14:59:50.589177133 +0800 CST m=+16.975543404 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:50.599 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:50.590 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:50.590 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_fast_fail/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_fast_fail/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_fast_fail/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_fast_fail/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_fast_fail/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2130e74c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-9tzt0, pid:1197, start at 2024-05-30 14:59:49.859147359 +0800 CST m=+22.407788191 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:49.870 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:49.843 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:49.843 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2130e74c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-9tzt0, pid:1197, start at 2024-05-30 14:59:49.859147359 +0800 CST m=+22.407788191 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:49.870 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:49.843 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:49.843 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2130e7540010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-9tzt0, pid:1272, start at 2024-05-30 14:59:49.8672215 +0800 CST m=+22.259594382 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:49.876 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:49.845 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:49.845 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/tiflash/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/tiflash/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/go-logr/logr v1.4.1 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading github.com/go-logr/stdr v1.2.2 table test.finish_mark not exists for 16-th check, retry later go: downloading github.com/DataDog/zstd v1.5.5 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/server_config_compatibility/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... go: downloading k8s.io/apimachinery v0.28.6 \033[0;36m<<< Run all test success >>>\033[0m go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading github.com/google/gofuzz v1.2.0 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd go: downloading k8s.io/klog/v2 v2.120.1 go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b nonempty select id, val from test.availability3 where id=1 and val=22 run task successfully empty select id, val from test.availability3 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later table multi_rocks.finish_mark not exists for 1-th check, retry later # diff Configuration. check-thread-count = 4 export-fix-sql = true check-struct-only = false [task] output-dir = "/tmp/tidb_cdc_test/consistent_replicate_nfs/sync_diff/output" source-instances = ["mysql1"] target-instance = "tidb0" target-check-tables = ["consistent_replicate_nfs.usertable*"] [data-sources] [data-sources.mysql1] host = "127.0.0.1" port = 4000 user = "root" password = "" snapshot = "450114978189934711" [data-sources.tidb0] host = "127.0.0.1" port = 3306 user = "root" password = "" [Thu May 30 14:59:53 CST 2024] <<<<<< START cdc server in big_txn case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.big_txn.26062608.out server --log-file /tmp/tidb_cdc_test/big_txn/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/big_txn/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' [2024/05/30 14:59:53.725 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2024/05/30 14:59:53.726 +08:00] [INFO] [apply.go:78] ["set memory limit"] [memoryLimit=10737418240] [2024/05/30 14:59:53.726 +08:00] [INFO] [apply.go:94] ["Start http pprof server"] [addr=:6060] [2024/05/30 14:59:53.726 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2024/05/30 14:59:53.726 +08:00] [INFO] [redo.go:167] ["apply redo log starts"] [checkpointTs=450114973497557060] [resolvedTs=450114978189934711] [2024/05/30 14:59:53.727 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=345.348ยตs] [2024/05/30 14:59:53.728 +08:00] [INFO] [tz.go:36] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/05/30 14:59:53.729 +08:00] [WARN] [config.go:394] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/05/30 14:59:53.729 +08:00] [ERROR] [config.go:209] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2024/05/30 14:59:53.735 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22NO_AUTO_VALUE_ON_ZERO%2CIGNORE_SPACE%2CONLY_FULL_GROUP_BY%2CALLOW_INVALID_DATES%2CNO_ENGINE_SUBSTITUTION%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/05/30 14:59:53.737 +08:00] [INFO] [mysql.go:198] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2024/05/30 14:59:53.739 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:59:53.739 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:59:53.739 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:59:53.739 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:59:53.739 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:59:53.739 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:59:53.739 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:59:53.739 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:59:53.739 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:59:53.739 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:59:53.739 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:59:53.740 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:59:53.740 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:59:53.740 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:59:53.740 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:59:53.740 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 14:59:53.740 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2024/05/30 14:59:53.740 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2024/05/30 14:59:53.740 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2024/05/30 14:59:53.740 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2024/05/30 14:59:53.740 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2024/05/30 14:59:53.740 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2024/05/30 14:59:53.740 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2024/05/30 14:59:53.740 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2024/05/30 14:59:53.740 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2024/05/30 14:59:53.740 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2024/05/30 14:59:53.740 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2024/05/30 14:59:53.740 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2024/05/30 14:59:53.740 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2024/05/30 14:59:53.740 +08:00] [INFO] [tz.go:36] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/05/30 14:59:53.740 +08:00] [WARN] [config.go:394] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/05/30 14:59:53.740 +08:00] [ERROR] [config.go:209] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2024/05/30 14:59:53.740 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2024/05/30 14:59:53.740 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2024/05/30 14:59:53.740 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2024/05/30 14:59:53.744 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22NO_ENGINE_SUBSTITUTION%2CNO_AUTO_VALUE_ON_ZERO%2CALLOW_INVALID_DATES%2CIGNORE_SPACE%2CONLY_FULL_GROUP_BY%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/05/30 14:59:53.746 +08:00] [INFO] [mysql_ddl_sink.go:99] ["MySQL DDL sink is created"] [namespace=] [changefeed=] [2024/05/30 14:59:53.746 +08:00] [INFO] [file.go:284] ["ignore logs which commitTs is greater than resolvedTs"] [filename=127.0.0.1:8300_a729ffa9-8852-4b3d-a5bd-178d5955033e_row_450114978189934713_9fafa409-5e39-418b-9d7b-b4dced733d04.log] [endTs=450114978189934711] [2024/05/30 14:59:53.775 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=48.505568ms] [2024/05/30 14:59:53.775 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.777 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.778 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.779 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.780 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.781 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.782 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.783 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.784 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.785 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.787 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.788 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.789 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.790 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.791 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.792 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.793 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.794 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.795 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.796 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.797 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.818 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.819 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.819 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.820 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.821 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.822 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.823 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.824 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.825 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.826 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.827 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.827 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.828 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.829 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.831 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.832 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.833 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.834 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.836 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.837 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.838 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.840 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.841 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.842 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.844 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.845 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.847 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.848 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.850 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.859 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.861 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.863 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.864 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.919 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.920 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.921 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.922 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.923 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.924 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.925 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.926 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.927 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc already exists) [Thu May 30 14:59:53 CST 2024] <<<<<< START cdc server in clustered_index case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.clustered_index.26582660.out server --log-file /tmp/tidb_cdc_test/clustered_index/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/clustered_index/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 5-th time, retry later [2024/05/30 14:59:53.929 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.930 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.930 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.932 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.932 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.933 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.934 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.935 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.936 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.937 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.938 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 14:59:53.939 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] table test.finish_mark exists + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_fast_fail.cli.2617.out cli tso query --pd=http://127.0.0.1:2379 check diff successfully + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tiflash.cli.2618.out cli tso query --pd=http://127.0.0.1:2379 empty select id, val from test.availability3 where id=1 run task successfully delete_owner_key pass wait process cdc.test exit for 1-th time... check diff failed 1-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/multi_source Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/server_config_compatibility Starting Upstream PD... table multi_rocks.finish_mark not exists for 2-th check, retry later Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... [2024/05/30 14:59:55.340 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}] [checkpointTs=450114978189934711] [2024/05/30 14:59:55.340 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}] [checkpointTs=450114978189934711] [2024/05/30 14:59:55.440 +08:00] [INFO] [redo.go:223] ["apply redo log finishes"] [appliedLogCount=19013] [appliedDDLCount=0] [currentCheckpoint=450114978189934711] [2024/05/30 14:59:55.441 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2024/05/30 14:59:55.441 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2024/05/30 14:59:55.441 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] [2024/05/30 14:59:55.441 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2024/05/30 14:59:55.441 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2024/05/30 14:59:55.441 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2024/05/30 14:59:55.441 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2024/05/30 14:59:55.441 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] [2024/05/30 14:59:55.441 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] [2024/05/30 14:59:55.441 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2024/05/30 14:59:55.441 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2024/05/30 14:59:55.441 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2024/05/30 14:59:55.441 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2024/05/30 14:59:55.441 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2024/05/30 14:59:55.441 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2024/05/30 14:59:55.441 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] Apply redo log successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... check diff successfully + set +x [Thu May 30 14:59:55 CST 2024] <<<<<< run test case consistent_replicate_nfs success! >>>>>> count(*) 45560 + tso='450114982865272833 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114982865272833 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + set +x + tso='450114982862651393 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114982862651393 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 14:59:55 CST 2024] <<<<<< START cdc server in tiflash case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tiflash.26422644.out server --log-file /tmp/tidb_cdc_test/tiflash/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/tiflash/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff successfully [Thu May 30 14:59:55 CST 2024] <<<<<< START cdc server in changefeed_fast_fail case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/InjectChangefeedFastFailError=return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_fast_fail.26592661.out server --log-file /tmp/tidb_cdc_test/changefeed_fast_fail/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_fast_fail/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... go: downloading github.com/jmespath/go-jmespath v0.4.0 wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 14:59:56 CST 2024] <<<<<< run test case many_pk_or_uk success! >>>>>> go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading github.com/google/s2a-go v0.1.7 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading github.com/felixge/httpsnoop v1.0.4 cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit run test case test_kill_capture [Thu May 30 14:59:56 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.51835185.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_kill_capture.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_kill_capture.server1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 2-th time, retry later [Thu May 30 14:59:57 CST 2024] <<<<<< START cdc server in bank case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bank.25962598.out server --log-file /tmp/tidb_cdc_test/bank/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/bank/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:56 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/00dd7900-5bf8-4db9-a6b5-7e48f233d710 {"id":"00dd7900-5bf8-4db9-a6b5-7e48f233d710","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bd18d3f 00dd7900-5bf8-4db9-a6b5-7e48f233d710 /tidb/cdc/default/default/upstream/7374683747123198612 {"id":7374683747123198612,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/00dd7900-5bf8-4db9-a6b5-7e48f233d710 {"id":"00dd7900-5bf8-4db9-a6b5-7e48f233d710","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bd18d3f 00dd7900-5bf8-4db9-a6b5-7e48f233d710 /tidb/cdc/default/default/upstream/7374683747123198612 {"id":7374683747123198612,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/00dd7900-5bf8-4db9-a6b5-7e48f233d710 {"id":"00dd7900-5bf8-4db9-a6b5-7e48f233d710","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bd18d3f 00dd7900-5bf8-4db9-a6b5-7e48f233d710 /tidb/cdc/default/default/upstream/7374683747123198612 {"id":7374683747123198612,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Thu May 30 14:59:56 CST 2024] <<<<<< START cdc server in big_txn case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.big_txn.26492651.out server --log-file /tmp/tidb_cdc_test/big_txn/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/big_txn/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:56 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/00dd7900-5bf8-4db9-a6b5-7e48f233d710 {"id":"00dd7900-5bf8-4db9-a6b5-7e48f233d710","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bd18d3f 00dd7900-5bf8-4db9-a6b5-7e48f233d710 /tidb/cdc/default/default/upstream/7374683747123198612 {"id":7374683747123198612,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/00dd7900-5bf8-4db9-a6b5-7e48f233d710 {"id":"00dd7900-5bf8-4db9-a6b5-7e48f233d710","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bd18d3f 00dd7900-5bf8-4db9-a6b5-7e48f233d710 /tidb/cdc/default/default/upstream/7374683747123198612 {"id":7374683747123198612,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/00dd7900-5bf8-4db9-a6b5-7e48f233d710 {"id":"00dd7900-5bf8-4db9-a6b5-7e48f233d710","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bd18d3f 00dd7900-5bf8-4db9-a6b5-7e48f233d710 /tidb/cdc/default/default/upstream/7374683747123198612 {"id":7374683747123198612,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.big_txn.cli.2678.out cli changefeed create '--sink-uri=mysql://normal:123456@127.0.0.1:3306?transaction-atomicity=none' + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:56 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b91f5a86-96f3-4767-bd36-fddee5ce4a1d {"id":"b91f5a86-96f3-4767-bd36-fddee5ce4a1d","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bcef94c b91f5a86-96f3-4767-bd36-fddee5ce4a1d /tidb/cdc/default/default/upstream/7374683751261433968 {"id":7374683751261433968,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b91f5a86-96f3-4767-bd36-fddee5ce4a1d {"id":"b91f5a86-96f3-4767-bd36-fddee5ce4a1d","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bcef94c b91f5a86-96f3-4767-bd36-fddee5ce4a1d /tidb/cdc/default/default/upstream/7374683751261433968 {"id":7374683751261433968,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b91f5a86-96f3-4767-bd36-fddee5ce4a1d {"id":"b91f5a86-96f3-4767-bd36-fddee5ce4a1d","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bcef94c b91f5a86-96f3-4767-bd36-fddee5ce4a1d /tidb/cdc/default/default/upstream/7374683751261433968 {"id":7374683751261433968,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: 1123868a-6b4f-4078-b012-55a75f223547 Info: {"upstream_id":7374683751261433968,"namespace":"default","id":"1123868a-6b4f-4078-b012-55a75f223547","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T14:59:57.002200683+08:00","start_ts":450114982692257795,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114982692257795,"checkpoint_ts":450114982692257795,"checkpoint_time":"2024-05-30 14:59:53.693"} Create changefeed successfully! ID: fc9f2115-d884-4988-be42-37fa9f8e9a69 Info: {"upstream_id":7374683747123198612,"namespace":"default","id":"fc9f2115-d884-4988-be42-37fa9f8e9a69","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306?transaction-atomicity=none","create_time":"2024-05-30T14:59:57.314505614+08:00","start_ts":450114983612645384,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"transaction_atomicity":"none","encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114983612645384,"checkpoint_ts":450114983612645384,"checkpoint_time":"2024-05-30 14:59:57.204"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table multi_rocks.finish_mark not exists for 3-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release check diff failed 1-th time, retry later [Thu May 30 14:59:58 CST 2024] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.77367738.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:58 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a77e7008-47d0-460e-9da3-1970262d6f55 {"id":"a77e7008-47d0-460e-9da3-1970262d6f55","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bd0895f a77e7008-47d0-460e-9da3-1970262d6f55 /tidb/cdc/default/default/upstream/7374683745518893039 {"id":7374683745518893039,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a77e7008-47d0-460e-9da3-1970262d6f55 {"id":"a77e7008-47d0-460e-9da3-1970262d6f55","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bd0895f a77e7008-47d0-460e-9da3-1970262d6f55 /tidb/cdc/default/default/upstream/7374683745518893039 {"id":7374683745518893039,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a77e7008-47d0-460e-9da3-1970262d6f55 {"id":"a77e7008-47d0-460e-9da3-1970262d6f55","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bd0895f a77e7008-47d0-460e-9da3-1970262d6f55 /tidb/cdc/default/default/upstream/7374683745518893039 {"id":7374683745518893039,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tiflash.cli.2693.out cli changefeed create --start-ts=450114982862651393 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:58 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2ecfeeb6-a3ab-418f-aa1e-22909bc7868f {"id":"2ecfeeb6-a3ab-418f-aa1e-22909bc7868f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bd6070b 2ecfeeb6-a3ab-418f-aa1e-22909bc7868f /tidb/cdc/default/default/upstream/7374683758811981208 {"id":7374683758811981208,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2ecfeeb6-a3ab-418f-aa1e-22909bc7868f {"id":"2ecfeeb6-a3ab-418f-aa1e-22909bc7868f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bd6070b 2ecfeeb6-a3ab-418f-aa1e-22909bc7868f /tidb/cdc/default/default/upstream/7374683758811981208 {"id":7374683758811981208,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2ecfeeb6-a3ab-418f-aa1e-22909bc7868f {"id":"2ecfeeb6-a3ab-418f-aa1e-22909bc7868f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bd6070b 2ecfeeb6-a3ab-418f-aa1e-22909bc7868f /tidb/cdc/default/default/upstream/7374683758811981208 {"id":7374683758811981208,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_fast_fail.cli.2707.out cli changefeed create --start-ts=450114982865272833 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' -c changefeed-fast-fail Create changefeed successfully! ID: d2561e8a-7ca6-40d2-b09f-0eae0bda8fee Info: {"upstream_id":7374683745518893039,"namespace":"default","id":"d2561e8a-7ca6-40d2-b09f-0eae0bda8fee","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T14:59:59.293976977+08:00","start_ts":450114982862651393,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114982862651393,"checkpoint_ts":450114982862651393,"checkpoint_time":"2024-05-30 14:59:54.343"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... Create changefeed successfully! ID: changefeed-fast-fail Info: {"upstream_id":7374683758811981208,"namespace":"default","id":"changefeed-fast-fail","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T14:59:59.419826669+08:00","start_ts":450114982865272833,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114982865272833,"checkpoint_ts":450114982865272833,"checkpoint_time":"2024-05-30 14:59:54.353"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... check diff successfully % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table multi_rocks.finish_mark not exists for 4-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ***************** properties ***************** "threadcount"="1" "readproportion"="0" "insertproportion"="0" "mysql.port"="4000" "mysql.db"="big_txn" "fieldcount"="100" "recordcount"="5000" "updateproportion"="0" "readallfields"="true" "scanproportion"="0" "operationcount"="0" "mysql.host"="127.0.0.1" "dotransactions"="false" "mysql.user"="root" "workload"="core" "requestdistribution"="uniform" ********************************************** + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:00:00 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0040cfa70 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114982857146378 MinTableBarrierTs:450114984167604236 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114982857146378, checkpointTs: 450114982857146378, state: Preparing span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114982857146378, checkpointTs: 450114982857146378, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114982857146378, checkpointTs: 450114982857146378, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7aca5dbc-8b2b-4822-a38d-9bca389bde33 {"id":"7aca5dbc-8b2b-4822-a38d-9bca389bde33","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525f48 7aca5dbc-8b2b-4822-a38d-9bca389bde33 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114982857146378,"min-table-barrier-ts":450114984167604236,"admin-job-type":0} /tidb/cdc/default/default/task/position/7aca5dbc-8b2b-4822-a38d-9bca389bde33/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0040cfa70 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114982857146378 MinTableBarrierTs:450114984167604236 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114982857146378, checkpointTs: 450114982857146378, state: Preparing span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114982857146378, checkpointTs: 450114982857146378, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114982857146378, checkpointTs: 450114982857146378, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7aca5dbc-8b2b-4822-a38d-9bca389bde33 {"id":"7aca5dbc-8b2b-4822-a38d-9bca389bde33","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525f48 7aca5dbc-8b2b-4822-a38d-9bca389bde33 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114982857146378,"min-table-barrier-ts":450114984167604236,"admin-job-type":0} /tidb/cdc/default/default/task/position/7aca5dbc-8b2b-4822-a38d-9bca389bde33/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0040cfa70 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114982857146378 MinTableBarrierTs:450114984167604236 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114982857146378, checkpointTs: 450114982857146378, state: Preparing span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114982857146378, checkpointTs: 450114982857146378, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114982857146378, checkpointTs: 450114982857146378, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7aca5dbc-8b2b-4822-a38d-9bca389bde33 {"id":"7aca5dbc-8b2b-4822-a38d-9bca389bde33","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525f48 7aca5dbc-8b2b-4822-a38d-9bca389bde33 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114982857146378,"min-table-barrier-ts":450114984167604236,"admin-job-type":0} /tidb/cdc/default/default/task/position/7aca5dbc-8b2b-4822-a38d-9bca389bde33/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:00:00 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/874a65dd-02db-4d18-aa42-e0506b1ee8a1 {"id":"874a65dd-02db-4d18-aa42-e0506b1ee8a1","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bc4130f 874a65dd-02db-4d18-aa42-e0506b1ee8a1 /tidb/cdc/default/default/upstream/7374683732008726242 {"id":7374683732008726242,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/874a65dd-02db-4d18-aa42-e0506b1ee8a1 {"id":"874a65dd-02db-4d18-aa42-e0506b1ee8a1","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bc4130f 874a65dd-02db-4d18-aa42-e0506b1ee8a1 /tidb/cdc/default/default/upstream/7374683732008726242 {"id":7374683732008726242,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/874a65dd-02db-4d18-aa42-e0506b1ee8a1 {"id":"874a65dd-02db-4d18-aa42-e0506b1ee8a1","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bc4130f 874a65dd-02db-4d18-aa42-e0506b1ee8a1 /tidb/cdc/default/default/upstream/7374683732008726242 {"id":7374683732008726242,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bank.cli.2650.out cli changefeed create --sink-uri=mysql://root@127.0.0.1:3306/ + set +x + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-fast-fail failed ErrStartTsBeforeGC + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-fast-fail + expected_state=failed + error_msg=ErrStartTsBeforeGC + tls_dir=ErrStartTsBeforeGC + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-fast-fail -s + info='{ "upstream_id": 7374683758811981208, "namespace": "default", "id": "changefeed-fast-fail", "state": "failed", "checkpoint_tso": 0, "checkpoint_time": "0001-01-01 00:00:00.000", "error": { "time": "2024-05-30T14:59:59.429213718+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError" } }' + echo '{ "upstream_id": 7374683758811981208, "namespace": "default", "id": "changefeed-fast-fail", "state": "failed", "checkpoint_tso": 0, "checkpoint_time": "0001-01-01 00:00:00.000", "error": { "time": "2024-05-30T14:59:59.429213718+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError" } }' { "upstream_id": 7374683758811981208, "namespace": "default", "id": "changefeed-fast-fail", "state": "failed", "checkpoint_tso": 0, "checkpoint_time": "0001-01-01 00:00:00.000", "error": { "time": "2024-05-30T14:59:59.429213718+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError" } } ++ echo '{' '"upstream_id":' 7374683758811981208, '"namespace":' '"default",' '"id":' '"changefeed-fast-fail",' '"state":' '"failed",' '"checkpoint_tso":' 0, '"checkpoint_time":' '"0001-01-01' '00:00:00.000",' '"error":' '{' '"time":' '"2024-05-30T14:59:59.429213718+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"' '}' '}' ++ jq -r .state + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ jq -r .error.message ++ echo '{' '"upstream_id":' 7374683758811981208, '"namespace":' '"default",' '"id":' '"changefeed-fast-fail",' '"state":' '"failed",' '"checkpoint_tso":' 0, '"checkpoint_time":' '"0001-01-01' '00:00:00.000",' '"error":' '{' '"time":' '"2024-05-30T14:59:59.429213718+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"' '}' '}' + message='[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError' + [[ ! [CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError =~ ErrStartTsBeforeGC ]] run task successfully Create changefeed successfully! ID: e6e72dba-98b9-465b-a92f-30de26e3ad86 Info: {"upstream_id":7374683732008726242,"namespace":"default","id":"e6e72dba-98b9-465b-a92f-30de26e3ad86","sink_uri":"mysql://root@127.0.0.1:3306/","create_time":"2024-05-30T15:00:00.907739939+08:00","start_ts":450114984503672835,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114984503672835,"checkpoint_ts":450114984503672835,"checkpoint_time":"2024-05-30 15:00:00.603"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table multi_rocks.finish_mark exists check diff successfully "is-owner": true, run task successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:00:01 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/296e9865-8722-4212-a273-7704f1994b9a {"id":"296e9865-8722-4212-a273-7704f1994b9a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bbf5b07 296e9865-8722-4212-a273-7704f1994b9a /tidb/cdc/default/default/upstream/7374683728060867366 {"id":7374683728060867366,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/296e9865-8722-4212-a273-7704f1994b9a {"id":"296e9865-8722-4212-a273-7704f1994b9a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bbf5b07 296e9865-8722-4212-a273-7704f1994b9a /tidb/cdc/default/default/upstream/7374683728060867366 {"id":7374683728060867366,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/296e9865-8722-4212-a273-7704f1994b9a {"id":"296e9865-8722-4212-a273-7704f1994b9a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bbf5b07 296e9865-8722-4212-a273-7704f1994b9a /tidb/cdc/default/default/upstream/7374683728060867366 {"id":7374683728060867366,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.cli.7783.out cli changefeed create --sink-uri=mysql://root@127.0.0.1:3306/ -c=ddl-only-block-related-table wait process cdc.test exit for 1-th time... Create changefeed successfully! ID: ddl-only-block-related-table Info: {"upstream_id":7374683728060867366,"namespace":"default","id":"ddl-only-block-related-table","sink_uri":"mysql://root@127.0.0.1:3306/","create_time":"2024-05-30T15:00:01.975774647+08:00","start_ts":450114984828993541,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114984828993541,"checkpoint_ts":450114984828993541,"checkpoint_time":"2024-05-30 15:00:01.844"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... wait process cdc.test exit for 1-th time... table clustered_index_test.t0 exists table clustered_index_test.t1 not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... table cdc_tiflash_test.multi_data_type exists check diff failed 1-th time, retry later check diff failed 2-th time, retry later + set +x + GO111MODULE=on + go run bank.go case.go -u 'root@tcp(127.0.0.1:4000)/bank' -d 'root@tcp(127.0.0.1:3306)/bank' --test-round=20000 -a 127.0.0.1:20080 wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading golang.org/x/sync v0.7.0 go: downloading github.com/spf13/cobra v1.8.0 go: downloading google.golang.org/grpc v1.64.0 owner pid: 5188 owner id 7aca5dbc-8b2b-4822-a38d-9bca389bde33 table test.availability1 exists nonempty select id, val from test.availability1 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading go.uber.org/atomic v1.11.0 go: downloading github.com/spf13/pflag v1.0.5 go: downloading go.uber.org/multierr v1.11.0 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 15:00:03 CST 2024] <<<<<< run test case changefeed_fast_fail success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:00:03 CST 2024] <<<<<< run test case multi_rocks success! >>>>>> go: downloading google.golang.org/protobuf v1.34.1 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240521202816-d264139d666e go: downloading golang.org/x/sys v0.20.0 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x table clustered_index_test.t1 exists table clustered_index_test.t2 not exists for 1-th check, retry later check diff successfully table ddl_only_block_related_table.finish_mark not exists for 1-th check, retry later wait process cdc.test exit for 1-th time... check diff failed 3-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... table clustered_index_test.t2 exists check table exists success check diff failed 1-th time, retry later nonempty select id, val from test.availability1 where id=1 and val=1 run task successfully [Thu May 30 15:00:04 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.53195321.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_kill_capture.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_kill_capture.server2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2131c7e0000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4168z, pid:6580, start at 2024-05-30 15:00:04.229600133 +0800 CST m=+4.969587552 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:04.235 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:04.216 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:04.216 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2131c7e0000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4168z, pid:6580, start at 2024-05-30 15:00:04.229600133 +0800 CST m=+4.969587552 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:04.235 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:04.216 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:04.216 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2131ca200016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4168z, pid:6660, start at 2024-05-30 15:00:04.395620169 +0800 CST m=+5.063783170 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:04.402 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:04.360 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:04.360 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_source/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_source/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/multi_source/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/multi_source/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_source/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 3-th time... table ddl_only_block_related_table.finish_mark not exists for 2-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:00:06 CST 2024] <<<<<< run test case tiflash success! >>>>>> check diff failed 4-th time, retry later + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_source.cli.7991.out cli tso query --pd=http://127.0.0.1:2379 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2131dd5c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-th7j8, pid:10190, start at 2024-05-30 15:00:05.616434942 +0800 CST m=+6.132482009 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:05.623 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:05.591 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:05.591 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2131dd5c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-th7j8, pid:10190, start at 2024-05-30 15:00:05.616434942 +0800 CST m=+6.132482009 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:05.623 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:05.591 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:05.591 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2131de280011 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-th7j8, pid:10276, start at 2024-05-30 15:00:05.669296259 +0800 CST m=+6.084267101 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:05.677 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:05.642 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:05.642 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/server_config_compatibility/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/server_config_compatibility/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/server_config_compatibility/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/server_config_compatibility/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/server_config_compatibility/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 2-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:00:08 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003a9e5a0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114986277863425 MinTableBarrierTs:450114986277863425 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2ae4e304-9fad-4784-8aa6-f81b35d30dec {"id":"2ae4e304-9fad-4784-8aa6-f81b35d30dec","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/7aca5dbc-8b2b-4822-a38d-9bca389bde33 {"id":"7aca5dbc-8b2b-4822-a38d-9bca389bde33","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525f48 7aca5dbc-8b2b-4822-a38d-9bca389bde33 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525f92 2ae4e304-9fad-4784-8aa6-f81b35d30dec /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114986277863425,"min-table-barrier-ts":450114986277863425,"admin-job-type":0} /tidb/cdc/default/default/task/position/2ae4e304-9fad-4784-8aa6-f81b35d30dec/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/7aca5dbc-8b2b-4822-a38d-9bca389bde33/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003a9e5a0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114986277863425 MinTableBarrierTs:450114986277863425 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2ae4e304-9fad-4784-8aa6-f81b35d30dec {"id":"2ae4e304-9fad-4784-8aa6-f81b35d30dec","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/7aca5dbc-8b2b-4822-a38d-9bca389bde33 {"id":"7aca5dbc-8b2b-4822-a38d-9bca389bde33","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525f48 7aca5dbc-8b2b-4822-a38d-9bca389bde33 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525f92 2ae4e304-9fad-4784-8aa6-f81b35d30dec /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114986277863425,"min-table-barrier-ts":450114986277863425,"admin-job-type":0} /tidb/cdc/default/default/task/position/2ae4e304-9fad-4784-8aa6-f81b35d30dec/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/7aca5dbc-8b2b-4822-a38d-9bca389bde33/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003a9e5a0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114986277863425 MinTableBarrierTs:450114986277863425 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2ae4e304-9fad-4784-8aa6-f81b35d30dec {"id":"2ae4e304-9fad-4784-8aa6-f81b35d30dec","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/7aca5dbc-8b2b-4822-a38d-9bca389bde33 {"id":"7aca5dbc-8b2b-4822-a38d-9bca389bde33","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525f48 7aca5dbc-8b2b-4822-a38d-9bca389bde33 /tidb/cdc/default/__cdc_meta__/owner/22318fc849525f92 2ae4e304-9fad-4784-8aa6-f81b35d30dec /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114986277863425,"min-table-barrier-ts":450114986277863425,"admin-job-type":0} /tidb/cdc/default/default/task/position/2ae4e304-9fad-4784-8aa6-f81b35d30dec/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/7aca5dbc-8b2b-4822-a38d-9bca389bde33/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep -v "7aca5dbc-8b2b-4822-a38d-9bca389bde33" | grep id table ddl_only_block_related_table.finish_mark not exists for 3-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/capture_session_done_during_task/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... "cluster-id": "default" "id": "2ae4e304-9fad-4784-8aa6-f81b35d30dec", "cluster-id": "default" run task successfully check diff successfully =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/resourcecontrol/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... check diff successfully wait process cdc.test exit for 1-th time... + set +x + tso='450114986420731905 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114986420731905 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 15:00:09 CST 2024] <<<<<< START cdc server in multi_source case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_source.80308032.out server --log-file /tmp/tidb_cdc_test/multi_source/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_source/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.server_config_compatibility.cli.11592.out cli tso query --pd=http://127.0.0.1:2379 wait process cdc.test exit for 1-th time... INSERT - Takes(s): 10.0, Count: 2236, OPS: 223.7, Avg(us): 4126, Min(us): 1360, Max(us): 88865, 95th(us): 4000, 99th(us): 81000 check diff failed 3-th time, retry later wait process cdc.test exit for 2-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... table ddl_only_block_related_table.finish_mark not exists for 4-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/capture_session_done_during_task Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... wait process cdc.test exit for 3-th time... nonempty select id, val from test.availability1 where id=2 and val=2 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:00:11 CST 2024] <<<<<< run test case changefeed_pause_resume success! >>>>>> cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:00:11 CST 2024] <<<<<< run test case owner_resign success! >>>>>> + set +x + tso='450114987030478849 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114987030478849 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x check diff failed 4-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/resourcecontrol Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... [Thu May 30 15:00:12 CST 2024] <<<<<< START cdc server in server_config_compatibility case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.server_config_compatibility.1164111643.out server --log-file /tmp/tidb_cdc_test/server_config_compatibility/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/server_config_compatibility/cdc_data --cluster-id default --config /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/server_config_compatibility/conf/server.toml + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 build mysql_test successfully run all mysql test cases time="2024-05-30T15:00:07+08:00" level=info msg="running tests: [alter_table bigint composite_index concurrent_ddl date_formats drop mysql_replace]" time="2024-05-30T15:00:07+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc000126000 {{} {} 0} {0 0} [0xc000314000] map[] 0 1 0xc00012a000 false map[0xc000314000:map[0xc000314000:true]] map[] 0 0 0 0 0 0 0 0 0x491680}" Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:00:12 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/afdfbb05-aeec-42a2-982c-daba22dc1198 {"id":"afdfbb05-aeec-42a2-982c-daba22dc1198","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c4ca6bd afdfbb05-aeec-42a2-982c-daba22dc1198 /tidb/cdc/default/default/upstream/7374683888161866607 {"id":7374683888161866607,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/afdfbb05-aeec-42a2-982c-daba22dc1198 {"id":"afdfbb05-aeec-42a2-982c-daba22dc1198","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c4ca6bd afdfbb05-aeec-42a2-982c-daba22dc1198 /tidb/cdc/default/default/upstream/7374683888161866607 {"id":7374683888161866607,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/afdfbb05-aeec-42a2-982c-daba22dc1198 {"id":"afdfbb05-aeec-42a2-982c-daba22dc1198","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c4ca6bd afdfbb05-aeec-42a2-982c-daba22dc1198 /tidb/cdc/default/default/upstream/7374683888161866607 {"id":7374683888161866607,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_source.cli.8077.out cli changefeed create --start-ts=450114986420731905 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: 5c1d62a1-ae45-4144-90ed-b4e04cfedf0e Info: {"upstream_id":7374683888161866607,"namespace":"default","id":"5c1d62a1-ae45-4144-90ed-b4e04cfedf0e","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T15:00:12.790094813+08:00","start_ts":450114986420731905,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114986420731905,"checkpoint_ts":450114986420731905,"checkpoint_time":"2024-05-30 15:00:07.916"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table ddl_only_block_related_table.finish_mark exists nonempty select id, val from test.availability1 where id=2 and val=2 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 2-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/batch_update_to_no_batch/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... wait process 7741 exit for 1-th time... \033[0;36m<<< Run all test success >>>\033[0m Verifying downstream PD is started... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/consistent_replicate_storage_file/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... wait process 7741 exit for 2-th time... check diff failed 5-th time, retry later Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + set +x go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/BurntSushi/toml v1.3.2 go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20240527072219-a97e6464c01d go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20240527072219-a97e6464c01d go: downloading github.com/pingcap/failpoint v0.0.0-20240412033321-fd0796e60f86 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading go.uber.org/atomic v1.11.0 start tidb cluster in /tmp/tidb_cdc_test/consistent_replicate_storage_file Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... wait process 7741 exit for 3-th time... go: downloading go.uber.org/multierr v1.11.0 /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (7741) - No such process wait process 7741 exit for 4-th time... process 7741 already exit [Thu May 30 15:00:14 CST 2024] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteNotDone=return(true)' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.78587860.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ./t/alter_table.test: ok! 70 test cases passed, take time 6.839002867 s time="2024-05-30T15:00:14+08:00" level=info msg="run test [alter_table] ok" time="2024-05-30T15:00:14+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc000308420 {{} {} 0} {0 0} [0xc0000fecf0] map[] 0 1 0xc000300ba0 false map[0xc0000fecf0:map[0xc0000fecf0:true]] map[] 0 0 0 0 0 0 0 0 0x491680}" [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc already exists) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:00:15 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bf409b8a-bac8-4ea1-a5e7-2a75edbdb8e7 {"id":"bf409b8a-bac8-4ea1-a5e7-2a75edbdb8e7","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c4f79e0 bf409b8a-bac8-4ea1-a5e7-2a75edbdb8e7 /tidb/cdc/default/default/upstream/7374683888803796551 {"id":7374683888803796551,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bf409b8a-bac8-4ea1-a5e7-2a75edbdb8e7 {"id":"bf409b8a-bac8-4ea1-a5e7-2a75edbdb8e7","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c4f79e0 bf409b8a-bac8-4ea1-a5e7-2a75edbdb8e7 /tidb/cdc/default/default/upstream/7374683888803796551 {"id":7374683888803796551,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bf409b8a-bac8-4ea1-a5e7-2a75edbdb8e7 {"id":"bf409b8a-bac8-4ea1-a5e7-2a75edbdb8e7","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c4f79e0 bf409b8a-bac8-4ea1-a5e7-2a75edbdb8e7 /tidb/cdc/default/default/upstream/7374683888803796551 {"id":7374683888803796551,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.server_config_compatibility.cli.11685.out cli changefeed create --start-ts=450114987030478849 --sink-uri=mysql+ssl://normal:123456@127.0.0.1:3306/ [Pipeline] // cache Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Create changefeed successfully! ID: 65acded3-13a8-444b-ba9b-bdd69f2c4116 Info: {"upstream_id":7374683888803796551,"namespace":"default","id":"65acded3-13a8-444b-ba9b-bdd69f2c4116","sink_uri":"mysql+ssl://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T15:00:15.589760013+08:00","start_ts":450114987030478849,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114987030478849,"checkpoint_ts":450114987030478849,"checkpoint_time":"2024-05-30 15:00:10.242"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... check diff failed 6-th time, retry later go: downloading golang.org/x/text v0.15.0 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/danjacques/gofslock v0.0.0-20220131014315-6e321f4509c8 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/pingcap/kvproto v0.0.0-20240522024016-df42997c2c57 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading golang.org/x/sync v0.7.0 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20240510070134-e7f530c64790 go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/golang/protobuf v1.5.4 go: downloading github.com/pingcap/tipb v0.0.0-20240507090649-2bf6bb0cb996 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/tikv/pd/client v0.0.0-20240210135946-3488a653ddd9 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading google.golang.org/grpc v1.64.0 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/prometheus/client_golang v1.19.1 go: downloading github.com/google/uuid v1.6.0 go: downloading golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842 go: downloading github.com/shirou/gopsutil/v3 v3.24.4 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/cockroachdb/pebble v0.0.0-20220415182917-06c9d3be25b3 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading github.com/google/btree v1.1.2 go: downloading github.com/klauspost/compress v1.17.8 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.1 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 start tidb cluster in /tmp/tidb_cdc_test/batch_update_to_no_batch Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... go: downloading github.com/aws/aws-sdk-go v1.44.259 go: downloading cloud.google.com/go v0.112.2 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.6 go: downloading google.golang.org/api v0.170.0 go: downloading golang.org/x/oauth2 v0.18.0 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading golang.org/x/sys v0.20.0 go: downloading golang.org/x/time v0.5.0 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading github.com/gorilla/mux v1.8.0 go: downloading github.com/gogo/protobuf v1.3.2 Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/region_merge/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/stretchr/testify v1.9.0 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/iancoleman/strcase v0.2.0 go: downloading github.com/prometheus/common v0.53.0 go: downloading github.com/golang/snappy v0.0.4 go: downloading google.golang.org/protobuf v1.34.1 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.0 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading golang.org/x/net v0.25.0 + set +x go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.1 nonempty select id, val from test.availability1 where id=2 and val=2 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 3-th time, retry later ERROR 1146 (42S02) at line 1: Table 'test.simple1' doesn't exist check data failed 1-th time, retry later go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.1 go: downloading golang.org/x/crypto v0.23.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/prometheus/procfs v0.15.0 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/sasha-s/go-deadlock v0.3.1 go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 go: downloading github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/json-iterator/go v1.1.12 Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240521202816-d264139d666e go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/petermattis/goid v0.0.0-20231207134359-e60b3f734c67 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/fatih/color v1.16.0 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/mattn/go-colorable v0.1.13 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:00:17 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table processor is not initialized *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0ea0152a-f73f-4e8e-8d8c-a37d9ac21b3b {"id":"0ea0152a-f73f-4e8e-8d8c-a37d9ac21b3b","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bbf5bd6 0ea0152a-f73f-4e8e-8d8c-a37d9ac21b3b /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7374683728060867366,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2024-05-30T15:00:01.975774647+08:00","start-ts":450114984828993541,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114984854945799} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":450114985510305839,"min-table-barrier-ts":450114987384897540,"admin-job-type":0} /tidb/cdc/default/default/task/position/0ea0152a-f73f-4e8e-8d8c-a37d9ac21b3b/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683728060867366 {"id":7374683728060867366,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table processor is not initialized *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0ea0152a-f73f-4e8e-8d8c-a37d9ac21b3b {"id":"0ea0152a-f73f-4e8e-8d8c-a37d9ac21b3b","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bbf5bd6 0ea0152a-f73f-4e8e-8d8c-a37d9ac21b3b /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7374683728060867366,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2024-05-30T15:00:01.975774647+08:00","start-ts":450114984828993541,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114984854945799} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":450114985510305839,"min-table-barrier-ts":450114987384897540,"admin-job-type":0} /tidb/cdc/default/default/task/position/0ea0152a-f73f-4e8e-8d8c-a37d9ac21b3b/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683728060867366 + grep -q 'failed to get info:' {"id":7374683728060867366,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table processor is not initialized *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0ea0152a-f73f-4e8e-8d8c-a37d9ac21b3b {"id":"0ea0152a-f73f-4e8e-8d8c-a37d9ac21b3b","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bbf5bd6 0ea0152a-f73f-4e8e-8d8c-a37d9ac21b3b /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7374683728060867366,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2024-05-30T15:00:01.975774647+08:00","start-ts":450114984828993541,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114984854945799} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":450114985510305839,"min-table-barrier-ts":450114987384897540,"admin-job-type":0} /tidb/cdc/default/default/task/position/0ea0152a-f73f-4e8e-8d8c-a37d9ac21b3b/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683728060867366 {"id":7374683728060867366,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x go: downloading github.com/mattn/go-runewidth v0.0.15 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/pingcap/badger v1.5.1-0.20230103063557-828f39b09b6d go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/robfig/cron v1.2.0 go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading github.com/DataDog/zstd v1.5.5 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/golang-jwt/jwt/v5 v5.2.0 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.12.0 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading github.com/go-logr/logr v1.4.1 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading github.com/go-logr/stdr v1.2.2 check diff failed 7-th time, retry later Verifying downstream PD is started... [2024/05/30 15:00:15.991 +08:00] [INFO] [case.go:445] ["open db success"] [dsn=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:00:15.993 +08:00] [INFO] [case.go:445] ["open db success"] [dsn=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:00:15.993 +08:00] [INFO] [case.go:383] ["drop tables"] [table=accounts_seq0] [2024/05/30 15:00:17.432 +08:00] [INFO] [case.go:365] ["insert 100 takes 651.009709ms"] [query="INSERT IGNORE INTO accounts_seq0 (id, counter, sequence, startts) VALUES (0, 0, 0, 0),(1, 0, 0, 0),(2, 0, 0, 0),(3, 0, 0, 0),(4, 0, 0, 0),(5, 0, 0, 0),(6, 0, 0, 0),(7, 0, 0, 0),(8, 0, 0, 0),(9, 0, 0, 0),(10, 0, 0, 0),(11, 0, 0, 0),(12, 0, 0, 0),(13, 0, 0, 0),(14, 0, 0, 0),(15, 0, 0, 0),(16, 0, 0, 0),(17, 0, 0, 0),(18, 0, 0, 0),(19, 0, 0, 0),(20, 0, 0, 0),(21, 0, 0, 0),(22, 0, 0, 0),(23, 0, 0, 0),(24, 0, 0, 0),(25, 0, 0, 0),(26, 0, 0, 0),(27, 0, 0, 0),(28, 0, 0, 0),(29, 0, 0, 0),(30, 0, 0, 0),(31, 0, 0, 0),(32, 0, 0, 0),(33, 0, 0, 0),(34, 0, 0, 0),(35, 0, 0, 0),(36, 0, 0, 0),(37, 0, 0, 0),(38, 0, 0, 0),(39, 0, 0, 0),(40, 0, 0, 0),(41, 0, 0, 0),(42, 0, 0, 0),(43, 0, 0, 0),(44, 0, 0, 0),(45, 0, 0, 0),(46, 0, 0, 0),(47, 0, 0, 0),(48, 0, 0, 0),(49, 0, 0, 0),(50, 0, 0, 0),(51, 0, 0, 0),(52, 0, 0, 0),(53, 0, 0, 0),(54, 0, 0, 0),(55, 0, 0, 0),(56, 0, 0, 0),(57, 0, 0, 0),(58, 0, 0, 0),(59, 0, 0, 0),(60, 0, 0, 0),(61, 0, 0, 0),(62, 0, 0, 0),(63, 0, 0, 0),(64, 0, 0, 0),(65, 0, 0, 0),(66, 0, 0, 0),(67, 0, 0, 0),(68, 0, 0, 0),(69, 0, 0, 0),(70, 0, 0, 0),(71, 0, 0, 0),(72, 0, 0, 0),(73, 0, 0, 0),(74, 0, 0, 0),(75, 0, 0, 0),(76, 0, 0, 0),(77, 0, 0, 0),(78, 0, 0, 0),(79, 0, 0, 0),(80, 0, 0, 0),(81, 0, 0, 0),(82, 0, 0, 0),(83, 0, 0, 0),(84, 0, 0, 0),(85, 0, 0, 0),(86, 0, 0, 0),(87, 0, 0, 0),(88, 0, 0, 0),(89, 0, 0, 0),(90, 0, 0, 0),(91, 0, 0, 0),(92, 0, 0, 0),(93, 0, 0, 0),(94, 0, 0, 0),(95, 0, 0, 0),(96, 0, 0, 0),(97, 0, 0, 0),(98, 0, 0, 0),(99, 0, 0, 0)"] [2024/05/30 15:00:17.432 +08:00] [INFO] [case.go:383] ["drop tables"] [table=accounts_seq1] [2024/05/30 15:00:18.088 +08:00] [INFO] [case.go:365] ["insert 100 takes 10.805709ms"] [query="INSERT IGNORE INTO accounts_seq1 (id, counter, sequence, startts) VALUES (0, 0, 0, 0),(1, 0, 0, 0),(2, 0, 0, 0),(3, 0, 0, 0),(4, 0, 0, 0),(5, 0, 0, 0),(6, 0, 0, 0),(7, 0, 0, 0),(8, 0, 0, 0),(9, 0, 0, 0),(10, 0, 0, 0),(11, 0, 0, 0),(12, 0, 0, 0),(13, 0, 0, 0),(14, 0, 0, 0),(15, 0, 0, 0),(16, 0, 0, 0),(17, 0, 0, 0),(18, 0, 0, 0),(19, 0, 0, 0),(20, 0, 0, 0),(21, 0, 0, 0),(22, 0, 0, 0),(23, 0, 0, 0),(24, 0, 0, 0),(25, 0, 0, 0),(26, 0, 0, 0),(27, 0, 0, 0),(28, 0, 0, 0),(29, 0, 0, 0),(30, 0, 0, 0),(31, 0, 0, 0),(32, 0, 0, 0),(33, 0, 0, 0),(34, 0, 0, 0),(35, 0, 0, 0),(36, 0, 0, 0),(37, 0, 0, 0),(38, 0, 0, 0),(39, 0, 0, 0),(40, 0, 0, 0),(41, 0, 0, 0),(42, 0, 0, 0),(43, 0, 0, 0),(44, 0, 0, 0),(45, 0, 0, 0),(46, 0, 0, 0),(47, 0, 0, 0),(48, 0, 0, 0),(49, 0, 0, 0),(50, 0, 0, 0),(51, 0, 0, 0),(52, 0, 0, 0),(53, 0, 0, 0),(54, 0, 0, 0),(55, 0, 0, 0),(56, 0, 0, 0),(57, 0, 0, 0),(58, 0, 0, 0),(59, 0, 0, 0),(60, 0, 0, 0),(61, 0, 0, 0),(62, 0, 0, 0),(63, 0, 0, 0),(64, 0, 0, 0),(65, 0, 0, 0),(66, 0, 0, 0),(67, 0, 0, 0),(68, 0, 0, 0),(69, 0, 0, 0),(70, 0, 0, 0),(71, 0, 0, 0),(72, 0, 0, 0),(73, 0, 0, 0),(74, 0, 0, 0),(75, 0, 0, 0),(76, 0, 0, 0),(77, 0, 0, 0),(78, 0, 0, 0),(79, 0, 0, 0),(80, 0, 0, 0),(81, 0, 0, 0),(82, 0, 0, 0),(83, 0, 0, 0),(84, 0, 0, 0),(85, 0, 0, 0),(86, 0, 0, 0),(87, 0, 0, 0),(88, 0, 0, 0),(89, 0, 0, 0),(90, 0, 0, 0),(91, 0, 0, 0),(92, 0, 0, 0),(93, 0, 0, 0),(94, 0, 0, 0),(95, 0, 0, 0),(96, 0, 0, 0),(97, 0, 0, 0),(98, 0, 0, 0),(99, 0, 0, 0)"] [2024/05/30 15:00:18.088 +08:00] [INFO] [case.go:383] ["drop tables"] [table=accounts_seq2] [2024/05/30 15:00:18.185 +08:00] [INFO] [case.go:365] ["insert 100 takes 7.738948ms"] [query="INSERT IGNORE INTO accounts_seq2 (id, counter, sequence, startts) VALUES (0, 0, 0, 0),(1, 0, 0, 0),(2, 0, 0, 0),(3, 0, 0, 0),(4, 0, 0, 0),(5, 0, 0, 0),(6, 0, 0, 0),(7, 0, 0, 0),(8, 0, 0, 0),(9, 0, 0, 0),(10, 0, 0, 0),(11, 0, 0, 0),(12, 0, 0, 0),(13, 0, 0, 0),(14, 0, 0, 0),(15, 0, 0, 0),(16, 0, 0, 0),(17, 0, 0, 0),(18, 0, 0, 0),(19, 0, 0, 0),(20, 0, 0, 0),(21, 0, 0, 0),(22, 0, 0, 0),(23, 0, 0, 0),(24, 0, 0, 0),(25, 0, 0, 0),(26, 0, 0, 0),(27, 0, 0, 0),(28, 0, 0, 0),(29, 0, 0, 0),(30, 0, 0, 0),(31, 0, 0, 0),(32, 0, 0, 0),(33, 0, 0, 0),(34, 0, 0, 0),(35, 0, 0, 0),(36, 0, 0, 0),(37, 0, 0, 0),(38, 0, 0, 0),(39, 0, 0, 0),(40, 0, 0, 0),(41, 0, 0, 0),(42, 0, 0, 0),(43, 0, 0, 0),(44, 0, 0, 0),(45, 0, 0, 0),(46, 0, 0, 0),(47, 0, 0, 0),(48, 0, 0, 0),(49, 0, 0, 0),(50, 0, 0, 0),(51, 0, 0, 0),(52, 0, 0, 0),(53, 0, 0, 0),(54, 0, 0, 0),(55, 0, 0, 0),(56, 0, 0, 0),(57, 0, 0, 0),(58, 0, 0, 0),(59, 0, 0, 0),(60, 0, 0, 0),(61, 0, 0, 0),(62, 0, 0, 0),(63, 0, 0, 0),(64, 0, 0, 0),(65, 0, 0, 0),(66, 0, 0, 0),(67, 0, 0, 0),(68, 0, 0, 0),(69, 0, 0, 0),(70, 0, 0, 0),(71, 0, 0, 0),(72, 0, 0, 0),(73, 0, 0, 0),(74, 0, 0, 0),(75, 0, 0, 0),(76, 0, 0, 0),(77, 0, 0, 0),(78, 0, 0, 0),(79, 0, 0, 0),(80, 0, 0, 0),(81, 0, 0, 0),(82, 0, 0, 0),(83, 0, 0, 0),(84, 0, 0, 0),(85, 0, 0, 0),(86, 0, 0, 0),(87, 0, 0, 0),(88, 0, 0, 0),(89, 0, 0, 0),(90, 0, 0, 0),(91, 0, 0, 0),(92, 0, 0, 0),(93, 0, 0, 0),(94, 0, 0, 0),(95, 0, 0, 0),(96, 0, 0, 0),(97, 0, 0, 0),(98, 0, 0, 0),(99, 0, 0, 0)"] [2024/05/30 15:00:18.185 +08:00] [INFO] [case.go:383] ["drop tables"] [table=accounts_seq3] [2024/05/30 15:00:18.283 +08:00] [INFO] [case.go:365] ["insert 100 takes 9.014526ms"] [query="INSERT IGNORE INTO accounts_seq3 (id, counter, sequence, startts) VALUES (0, 0, 0, 0),(1, 0, 0, 0),(2, 0, 0, 0),(3, 0, 0, 0),(4, 0, 0, 0),(5, 0, 0, 0),(6, 0, 0, 0),(7, 0, 0, 0),(8, 0, 0, 0),(9, 0, 0, 0),(10, 0, 0, 0),(11, 0, 0, 0),(12, 0, 0, 0),(13, 0, 0, 0),(14, 0, 0, 0),(15, 0, 0, 0),(16, 0, 0, 0),(17, 0, 0, 0),(18, 0, 0, 0),(19, 0, 0, 0),(20, 0, 0, 0),(21, 0, 0, 0),(22, 0, 0, 0),(23, 0, 0, 0),(24, 0, 0, 0),(25, 0, 0, 0),(26, 0, 0, 0),(27, 0, 0, 0),(28, 0, 0, 0),(29, 0, 0, 0),(30, 0, 0, 0),(31, 0, 0, 0),(32, 0, 0, 0),(33, 0, 0, 0),(34, 0, 0, 0),(35, 0, 0, 0),(36, 0, 0, 0),(37, 0, 0, 0),(38, 0, 0, 0),(39, 0, 0, 0),(40, 0, 0, 0),(41, 0, 0, 0),(42, 0, 0, 0),(43, 0, 0, 0),(44, 0, 0, 0),(45, 0, 0, 0),(46, 0, 0, 0),(47, 0, 0, 0),(48, 0, 0, 0),(49, 0, 0, 0),(50, 0, 0, 0),(51, 0, 0, 0),(52, 0, 0, 0),(53, 0, 0, 0),(54, 0, 0, 0),(55, 0, 0, 0),(56, 0, 0, 0),(57, 0, 0, 0),(58, 0, 0, 0),(59, 0, 0, 0),(60, 0, 0, 0),(61, 0, 0, 0),(62, 0, 0, 0),(63, 0, 0, 0),(64, 0, 0, 0),(65, 0, 0, 0),(66, 0, 0, 0),(67, 0, 0, 0),(68, 0, 0, 0),(69, 0, 0, 0),(70, 0, 0, 0),(71, 0, 0, 0),(72, 0, 0, 0),(73, 0, 0, 0),(74, 0, 0, 0),(75, 0, 0, 0),(76, 0, 0, 0),(77, 0, 0, 0),(78, 0, 0, 0),(79, 0, 0, 0),(80, 0, 0, 0),(81, 0, 0, 0),(82, 0, 0, 0),(83, 0, 0, 0),(84, 0, 0, 0),(85, 0, 0, 0),(86, 0, 0, 0),(87, 0, 0, 0),(88, 0, 0, 0),(89, 0, 0, 0),(90, 0, 0, 0),(91, 0, 0, 0),(92, 0, 0, 0),(93, 0, 0, 0),(94, 0, 0, 0),(95, 0, 0, 0),(96, 0, 0, 0),(97, 0, 0, 0),(98, 0, 0, 0),(99, 0, 0, 0)"] [2024/05/30 15:00:18.283 +08:00] [INFO] [case.go:383] ["drop tables"] [table=accounts_seq4] [2024/05/30 15:00:18.380 +08:00] [INFO] [case.go:365] ["insert 100 takes 8.771931ms"] [query="INSERT IGNORE INTO accounts_seq4 (id, counter, sequence, startts) VALUES (0, 0, 0, 0),(1, 0, 0, 0),(2, 0, 0, 0),(3, 0, 0, 0),(4, 0, 0, 0),(5, 0, 0, 0),(6, 0, 0, 0),(7, 0, 0, 0),(8, 0, 0, 0),(9, 0, 0, 0),(10, 0, 0, 0),(11, 0, 0, 0),(12, 0, 0, 0),(13, 0, 0, 0),(14, 0, 0, 0),(15, 0, 0, 0),(16, 0, 0, 0),(17, 0, 0, 0),(18, 0, 0, 0),(19, 0, 0, 0),(20, 0, 0, 0),(21, 0, 0, 0),(22, 0, 0, 0),(23, 0, 0, 0),(24, 0, 0, 0),(25, 0, 0, 0),(26, 0, 0, 0),(27, 0, 0, 0),(28, 0, 0, 0),(29, 0, 0, 0),(30, 0, 0, 0),(31, 0, 0, 0),(32, 0, 0, 0),(33, 0, 0, 0),(34, 0, 0, 0),(35, 0, 0, 0),(36, 0, 0, 0),(37, 0, 0, 0),(38, 0, 0, 0),(39, 0, 0, 0),(40, 0, 0, 0),(41, 0, 0, 0),(42, 0, 0, 0),(43, 0, 0, 0),(44, 0, 0, 0),(45, 0, 0, 0),(46, 0, 0, 0),(47, 0, 0, 0),(48, 0, 0, 0),(49, 0, 0, 0),(50, 0, 0, 0),(51, 0, 0, 0),(52, 0, 0, 0),(53, 0, 0, 0),(54, 0, 0, 0),(55, 0, 0, 0),(56, 0, 0, 0),(57, 0, 0, 0),(58, 0, 0, 0),(59, 0, 0, 0),(60, 0, 0, 0),(61, 0, 0, 0),(62, 0, 0, 0),(63, 0, 0, 0),(64, 0, 0, 0),(65, 0, 0, 0),(66, 0, 0, 0),(67, 0, 0, 0),(68, 0, 0, 0),(69, 0, 0, 0),(70, 0, 0, 0),(71, 0, 0, 0),(72, 0, 0, 0),(73, 0, 0, 0),(74, 0, 0, 0),(75, 0, 0, 0),(76, 0, 0, 0),(77, 0, 0, 0),(78, 0, 0, 0),(79, 0, 0, 0),(80, 0, 0, 0),(81, 0, 0, 0),(82, 0, 0, 0),(83, 0, 0, 0),(84, 0, 0, 0),(85, 0, 0, 0),(86, 0, 0, 0),(87, 0, 0, 0),(88, 0, 0, 0),(89, 0, 0, 0),(90, 0, 0, 0),(91, 0, 0, 0),(92, 0, 0, 0),(93, 0, 0, 0),(94, 0, 0, 0),(95, 0, 0, 0),(96, 0, 0, 0),(97, 0, 0, 0),(98, 0, 0, 0),(99, 0, 0, 0)"] [2024/05/30 15:00:18.380 +08:00] [INFO] [case.go:383] ["drop tables"] [table=accounts0] check diff failed 1-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:00:18.477 +08:00] [INFO] [case.go:365] ["insert 100 takes 9.028939ms"] [query="INSERT IGNORE INTO accounts0 (id, balance, startts) VALUES (0, 1000, 0),(1, 1000, 0),(2, 1000, 0),(3, 1000, 0),(4, 1000, 0),(5, 1000, 0),(6, 1000, 0),(7, 1000, 0),(8, 1000, 0),(9, 1000, 0),(10, 1000, 0),(11, 1000, 0),(12, 1000, 0),(13, 1000, 0),(14, 1000, 0),(15, 1000, 0),(16, 1000, 0),(17, 1000, 0),(18, 1000, 0),(19, 1000, 0),(20, 1000, 0),(21, 1000, 0),(22, 1000, 0),(23, 1000, 0),(24, 1000, 0),(25, 1000, 0),(26, 1000, 0),(27, 1000, 0),(28, 1000, 0),(29, 1000, 0),(30, 1000, 0),(31, 1000, 0),(32, 1000, 0),(33, 1000, 0),(34, 1000, 0),(35, 1000, 0),(36, 1000, 0),(37, 1000, 0),(38, 1000, 0),(39, 1000, 0),(40, 1000, 0),(41, 1000, 0),(42, 1000, 0),(43, 1000, 0),(44, 1000, 0),(45, 1000, 0),(46, 1000, 0),(47, 1000, 0),(48, 1000, 0),(49, 1000, 0),(50, 1000, 0),(51, 1000, 0),(52, 1000, 0),(53, 1000, 0),(54, 1000, 0),(55, 1000, 0),(56, 1000, 0),(57, 1000, 0),(58, 1000, 0),(59, 1000, 0),(60, 1000, 0),(61, 1000, 0),(62, 1000, 0),(63, 1000, 0),(64, 1000, 0),(65, 1000, 0),(66, 1000, 0),(67, 1000, 0),(68, 1000, 0),(69, 1000, 0),(70, 1000, 0),(71, 1000, 0),(72, 1000, 0),(73, 1000, 0),(74, 1000, 0),(75, 1000, 0),(76, 1000, 0),(77, 1000, 0),(78, 1000, 0),(79, 1000, 0),(80, 1000, 0),(81, 1000, 0),(82, 1000, 0),(83, 1000, 0),(84, 1000, 0),(85, 1000, 0),(86, 1000, 0),(87, 1000, 0),(88, 1000, 0),(89, 1000, 0),(90, 1000, 0),(91, 1000, 0),(92, 1000, 0),(93, 1000, 0),(94, 1000, 0),(95, 1000, 0),(96, 1000, 0),(97, 1000, 0),(98, 1000, 0),(99, 1000, 0)"] [2024/05/30 15:00:18.478 +08:00] [INFO] [case.go:383] ["drop tables"] [table=accounts1] [2024/05/30 15:00:18.572 +08:00] [INFO] [case.go:365] ["insert 100 takes 8.760335ms"] [query="INSERT IGNORE INTO accounts1 (id, balance, startts) VALUES (0, 1000, 0),(1, 1000, 0),(2, 1000, 0),(3, 1000, 0),(4, 1000, 0),(5, 1000, 0),(6, 1000, 0),(7, 1000, 0),(8, 1000, 0),(9, 1000, 0),(10, 1000, 0),(11, 1000, 0),(12, 1000, 0),(13, 1000, 0),(14, 1000, 0),(15, 1000, 0),(16, 1000, 0),(17, 1000, 0),(18, 1000, 0),(19, 1000, 0),(20, 1000, 0),(21, 1000, 0),(22, 1000, 0),(23, 1000, 0),(24, 1000, 0),(25, 1000, 0),(26, 1000, 0),(27, 1000, 0),(28, 1000, 0),(29, 1000, 0),(30, 1000, 0),(31, 1000, 0),(32, 1000, 0),(33, 1000, 0),(34, 1000, 0),(35, 1000, 0),(36, 1000, 0),(37, 1000, 0),(38, 1000, 0),(39, 1000, 0),(40, 1000, 0),(41, 1000, 0),(42, 1000, 0),(43, 1000, 0),(44, 1000, 0),(45, 1000, 0),(46, 1000, 0),(47, 1000, 0),(48, 1000, 0),(49, 1000, 0),(50, 1000, 0),(51, 1000, 0),(52, 1000, 0),(53, 1000, 0),(54, 1000, 0),(55, 1000, 0),(56, 1000, 0),(57, 1000, 0),(58, 1000, 0),(59, 1000, 0),(60, 1000, 0),(61, 1000, 0),(62, 1000, 0),(63, 1000, 0),(64, 1000, 0),(65, 1000, 0),(66, 1000, 0),(67, 1000, 0),(68, 1000, 0),(69, 1000, 0),(70, 1000, 0),(71, 1000, 0),(72, 1000, 0),(73, 1000, 0),(74, 1000, 0),(75, 1000, 0),(76, 1000, 0),(77, 1000, 0),(78, 1000, 0),(79, 1000, 0),(80, 1000, 0),(81, 1000, 0),(82, 1000, 0),(83, 1000, 0),(84, 1000, 0),(85, 1000, 0),(86, 1000, 0),(87, 1000, 0),(88, 1000, 0),(89, 1000, 0),(90, 1000, 0),(91, 1000, 0),(92, 1000, 0),(93, 1000, 0),(94, 1000, 0),(95, 1000, 0),(96, 1000, 0),(97, 1000, 0),(98, 1000, 0),(99, 1000, 0)"] [2024/05/30 15:00:18.572 +08:00] [INFO] [case.go:383] ["drop tables"] [table=accounts2] [2024/05/30 15:00:18.674 +08:00] [INFO] [case.go:365] ["insert 100 takes 9.185881ms"] [query="INSERT IGNORE INTO accounts2 (id, balance, startts) VALUES (0, 1000, 0),(1, 1000, 0),(2, 1000, 0),(3, 1000, 0),(4, 1000, 0),(5, 1000, 0),(6, 1000, 0),(7, 1000, 0),(8, 1000, 0),(9, 1000, 0),(10, 1000, 0),(11, 1000, 0),(12, 1000, 0),(13, 1000, 0),(14, 1000, 0),(15, 1000, 0),(16, 1000, 0),(17, 1000, 0),(18, 1000, 0),(19, 1000, 0),(20, 1000, 0),(21, 1000, 0),(22, 1000, 0),(23, 1000, 0),(24, 1000, 0),(25, 1000, 0),(26, 1000, 0),(27, 1000, 0),(28, 1000, 0),(29, 1000, 0),(30, 1000, 0),(31, 1000, 0),(32, 1000, 0),(33, 1000, 0),(34, 1000, 0),(35, 1000, 0),(36, 1000, 0),(37, 1000, 0),(38, 1000, 0),(39, 1000, 0),(40, 1000, 0),(41, 1000, 0),(42, 1000, 0),(43, 1000, 0),(44, 1000, 0),(45, 1000, 0),(46, 1000, 0),(47, 1000, 0),(48, 1000, 0),(49, 1000, 0),(50, 1000, 0),(51, 1000, 0),(52, 1000, 0),(53, 1000, 0),(54, 1000, 0),(55, 1000, 0),(56, 1000, 0),(57, 1000, 0),(58, 1000, 0),(59, 1000, 0),(60, 1000, 0),(61, 1000, 0),(62, 1000, 0),(63, 1000, 0),(64, 1000, 0),(65, 1000, 0),(66, 1000, 0),(67, 1000, 0),(68, 1000, 0),(69, 1000, 0),(70, 1000, 0),(71, 1000, 0),(72, 1000, 0),(73, 1000, 0),(74, 1000, 0),(75, 1000, 0),(76, 1000, 0),(77, 1000, 0),(78, 1000, 0),(79, 1000, 0),(80, 1000, 0),(81, 1000, 0),(82, 1000, 0),(83, 1000, 0),(84, 1000, 0),(85, 1000, 0),(86, 1000, 0),(87, 1000, 0),(88, 1000, 0),(89, 1000, 0),(90, 1000, 0),(91, 1000, 0),(92, 1000, 0),(93, 1000, 0),(94, 1000, 0),(95, 1000, 0),(96, 1000, 0),(97, 1000, 0),(98, 1000, 0),(99, 1000, 0)"] [2024/05/30 15:00:18.675 +08:00] [INFO] [case.go:383] ["drop tables"] [table=accounts3] ./t/bigint.test: ok! 138 test cases passed, take time 3.434521057 s time="2024-05-30T15:00:18+08:00" level=info msg="run test [bigint] ok" time="2024-05-30T15:00:18+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc00042c1f8 {{} {} 0} {0 0} [0xc000468990] map[] 0 1 0xc00003e7e0 false map[0xc000468990:map[0xc000468990:true]] map[] 0 0 0 0 0 0 0 0 0x491680}" ./t/composite_index.test: ok! 12 test cases passed, take time 0.273683596 s time="2024-05-30T15:00:18+08:00" level=info msg="run test [composite_index] ok" time="2024-05-30T15:00:18+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc0000ca100 {{} {} 0} {0 0} [0xc00028c090] map[] 0 1 0xc00038a240 false map[0xc00028c090:map[0xc00028c090:true]] map[] 0 0 0 0 0 0 0 0 0x491680}" [2024/05/30 15:00:06.713 +08:00] [WARN] [diff.go:179] ["table struct is not equal"] [reason="column num not equal, one is 34 another is 21"] [2024/05/30 15:00:11.714 +08:00] [WARN] [diff.go:179] ["table struct is not equal"] [reason="column num not equal, one is 34 another is 28"] [2024/05/30 15:00:18.770 +08:00] [INFO] [case.go:365] ["insert 100 takes 9.057973ms"] [query="INSERT IGNORE INTO accounts3 (id, balance, startts) VALUES (0, 1000, 0),(1, 1000, 0),(2, 1000, 0),(3, 1000, 0),(4, 1000, 0),(5, 1000, 0),(6, 1000, 0),(7, 1000, 0),(8, 1000, 0),(9, 1000, 0),(10, 1000, 0),(11, 1000, 0),(12, 1000, 0),(13, 1000, 0),(14, 1000, 0),(15, 1000, 0),(16, 1000, 0),(17, 1000, 0),(18, 1000, 0),(19, 1000, 0),(20, 1000, 0),(21, 1000, 0),(22, 1000, 0),(23, 1000, 0),(24, 1000, 0),(25, 1000, 0),(26, 1000, 0),(27, 1000, 0),(28, 1000, 0),(29, 1000, 0),(30, 1000, 0),(31, 1000, 0),(32, 1000, 0),(33, 1000, 0),(34, 1000, 0),(35, 1000, 0),(36, 1000, 0),(37, 1000, 0),(38, 1000, 0),(39, 1000, 0),(40, 1000, 0),(41, 1000, 0),(42, 1000, 0),(43, 1000, 0),(44, 1000, 0),(45, 1000, 0),(46, 1000, 0),(47, 1000, 0),(48, 1000, 0),(49, 1000, 0),(50, 1000, 0),(51, 1000, 0),(52, 1000, 0),(53, 1000, 0),(54, 1000, 0),(55, 1000, 0),(56, 1000, 0),(57, 1000, 0),(58, 1000, 0),(59, 1000, 0),(60, 1000, 0),(61, 1000, 0),(62, 1000, 0),(63, 1000, 0),(64, 1000, 0),(65, 1000, 0),(66, 1000, 0),(67, 1000, 0),(68, 1000, 0),(69, 1000, 0),(70, 1000, 0),(71, 1000, 0),(72, 1000, 0),(73, 1000, 0),(74, 1000, 0),(75, 1000, 0),(76, 1000, 0),(77, 1000, 0),(78, 1000, 0),(79, 1000, 0),(80, 1000, 0),(81, 1000, 0),(82, 1000, 0),(83, 1000, 0),(84, 1000, 0),(85, 1000, 0),(86, 1000, 0),(87, 1000, 0),(88, 1000, 0),(89, 1000, 0),(90, 1000, 0),(91, 1000, 0),(92, 1000, 0),(93, 1000, 0),(94, 1000, 0),(95, 1000, 0),(96, 1000, 0),(97, 1000, 0),(98, 1000, 0),(99, 1000, 0)"] [2024/05/30 15:00:18.770 +08:00] [INFO] [case.go:383] ["drop tables"] [table=accounts4] [2024/05/30 15:00:18.863 +08:00] [INFO] [case.go:365] ["insert 100 takes 8.97963ms"] [query="INSERT IGNORE INTO accounts4 (id, balance, startts) VALUES (0, 1000, 0),(1, 1000, 0),(2, 1000, 0),(3, 1000, 0),(4, 1000, 0),(5, 1000, 0),(6, 1000, 0),(7, 1000, 0),(8, 1000, 0),(9, 1000, 0),(10, 1000, 0),(11, 1000, 0),(12, 1000, 0),(13, 1000, 0),(14, 1000, 0),(15, 1000, 0),(16, 1000, 0),(17, 1000, 0),(18, 1000, 0),(19, 1000, 0),(20, 1000, 0),(21, 1000, 0),(22, 1000, 0),(23, 1000, 0),(24, 1000, 0),(25, 1000, 0),(26, 1000, 0),(27, 1000, 0),(28, 1000, 0),(29, 1000, 0),(30, 1000, 0),(31, 1000, 0),(32, 1000, 0),(33, 1000, 0),(34, 1000, 0),(35, 1000, 0),(36, 1000, 0),(37, 1000, 0),(38, 1000, 0),(39, 1000, 0),(40, 1000, 0),(41, 1000, 0),(42, 1000, 0),(43, 1000, 0),(44, 1000, 0),(45, 1000, 0),(46, 1000, 0),(47, 1000, 0),(48, 1000, 0),(49, 1000, 0),(50, 1000, 0),(51, 1000, 0),(52, 1000, 0),(53, 1000, 0),(54, 1000, 0),(55, 1000, 0),(56, 1000, 0),(57, 1000, 0),(58, 1000, 0),(59, 1000, 0),(60, 1000, 0),(61, 1000, 0),(62, 1000, 0),(63, 1000, 0),(64, 1000, 0),(65, 1000, 0),(66, 1000, 0),(67, 1000, 0),(68, 1000, 0),(69, 1000, 0),(70, 1000, 0),(71, 1000, 0),(72, 1000, 0),(73, 1000, 0),(74, 1000, 0),(75, 1000, 0),(76, 1000, 0),(77, 1000, 0),(78, 1000, 0),(79, 1000, 0),(80, 1000, 0),(81, 1000, 0),(82, 1000, 0),(83, 1000, 0),(84, 1000, 0),(85, 1000, 0),(86, 1000, 0),(87, 1000, 0),(88, 1000, 0),(89, 1000, 0),(90, 1000, 0),(91, 1000, 0),(92, 1000, 0),(93, 1000, 0),(94, 1000, 0),(95, 1000, 0),(96, 1000, 0),(97, 1000, 0),(98, 1000, 0),(99, 1000, 0)"] ./t/concurrent_ddl.test: ok! 8 test cases passed, take time 0.433261499 s time="2024-05-30T15:00:19+08:00" level=info msg="run test [concurrent_ddl] ok" time="2024-05-30T15:00:19+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc000126180 {{} {} 0} {0 0} [0xc000468b40] map[] 0 1 0xc00012a6c0 false map[0xc000468b40:map[0xc000468b40:true]] map[] 0 0 0 0 0 0 0 0 0x491680}" TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 2-th time, retry later Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore INSERT - Takes(s): 20.0, Count: 4582, OPS: 229.2, Avg(us): 4047, Min(us): 1273, Max(us): 88865, 95th(us): 4000, 99th(us): 81000 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 8-th time, retry later ./t/date_formats.test: ok! 99 test cases passed, take time 1.142228431 s time="2024-05-30T15:00:20+08:00" level=info msg="run test [date_formats] ok" time="2024-05-30T15:00:20+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc0001264a8 {{} {} 0} {0 0} [0xc0004c5290] map[] 0 1 0xc00012b5c0 false map[0xc0004c5290:map[0xc0004c5290:true]] map[] 0 0 0 0 0 0 0 0 0x491680}" Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully check_ts_not_forward ddl-only-block-related-table ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... [2024/05/30 15:00:20.957 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 3-th time, retry later Run finished, takes 21.766097415s INSERT - Takes(s): 21.8, Count: 5000, OPS: 229.8, Avg(us): 4053, Min(us): 1273, Max(us): 88865, 95th(us): 4000, 99th(us): 80000 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading github.com/jmespath/go-jmespath v0.4.0 go: downloading github.com/google/s2a-go v0.1.7 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading github.com/felixge/httpsnoop v1.0.4 table big_txn.usertable exists [Pipeline] } [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc already exists) check diff failed 9-th time, retry later [Pipeline] // cache [Pipeline] } ./t/drop.test: ok! 48 test cases passed, take time 2.108881349 s time="2024-05-30T15:00:22+08:00" level=info msg="run test [drop] ok" time="2024-05-30T15:00:22+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc000126168 {{} {} 0} {0 0} [0xc000382000] map[] 0 1 0xc00012a840 false map[0xc000382000:map[0xc000382000:true]] map[] 0 0 0 0 0 0 0 0 0x491680}" ./t/mysql_replace.test: ok! 16 test cases passed, take time 0.19402173 s time="2024-05-30T15:00:22+08:00" level=info msg="run test [mysql_replace] ok" Great, All tests passed mysqltest end table test.finish_mark not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2132dd580013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-lpxxn, pid:17745, start at 2024-05-30 15:00:22.000422438 +0800 CST m=+5.669777443 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:22.009 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:21.974 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:21.974 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2132dd580013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-lpxxn, pid:17745, start at 2024-05-30 15:00:22.000422438 +0800 CST m=+5.669777443 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:22.009 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:21.974 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:21.974 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2132dfb80014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-lpxxn, pid:17815, start at 2024-05-30 15:00:22.158334705 +0800 CST m=+5.745593971 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:22.166 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:22.126 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:22.126 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2024/05/30 15:00:22.970 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check data successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) nonempty select id, val from test.availability1 where id=2 and val=2 run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_hang_up_capture [Thu May 30 15:00:24 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.54965498.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_hang_up_capture.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_hang_up_capture.server1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 10-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/region_merge Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... table test.finish_mark not exists for 2-th check, retry later [Pipeline] // dir cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 15:00:24 CST 2024] <<<<<< run test case server_config_compatibility success! >>>>>> [Pipeline] // cache [Pipeline] } [2024/05/30 15:00:24.983 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] [Pipeline] } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // dir ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] } [Pipeline] // dir [Pipeline] // withCredentials =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/cli/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] } [Pipeline] } + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_session_done_during_task.cli.19079.out cli tso query --pd=http://127.0.0.1:2379 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/processor_etcd_worker_delay/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] // withCredentials [Pipeline] } check diff failed 11-th time, retry later [Pipeline] // timeout [Pipeline] // withCredentials [Pipeline] } [Pipeline] } [2024/05/30 15:00:26.996 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2133269c000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3, pid:9528, start at 2024-05-30 15:00:26.680810468 +0800 CST m=+7.229832685 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:26.688 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:26.663 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:26.663 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // timeout [Pipeline] } table test.finish_mark not exists for 3-th check, retry later [Pipeline] // stage [Pipeline] // timeout [Pipeline] } [Pipeline] } [Pipeline] // stage VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21330ab00011 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t5729, pid:9848, start at 2024-05-30 15:00:24.920632042 +0800 CST m=+7.573884078 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:24.936 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:24.925 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:24.925 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21330ab00011 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t5729, pid:9848, start at 2024-05-30 15:00:24.920632042 +0800 CST m=+7.573884078 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:24.936 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:24.925 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:24.925 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21330bb00021 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t5729, pid:9872, start at 2024-05-30 15:00:24.966109857 +0800 CST m=+7.484402517 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:24.981 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:24.989 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:24.989 +0800 All versions after safe point can be accessed. (DO NOT EDIT) [Pipeline] } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213327940014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-cvg08, pid:3657, start at 2024-05-30 15:00:26.763700242 +0800 CST m=+5.595651039 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:26.771 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:26.773 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:26.773 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // container [Pipeline] // stage [Pipeline] } [Pipeline] } + (( i++ )) + (( i <= 50 )) Starting Upstream TiFlash... TiFlash ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info Release Version: v7.5.1-26-gdb0862135 Edition: Community * About to connect() to 127.0.0.1 port 8300 (#0) Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD * Trying 127.0.0.1... UTC Build Time: 2024-05-27 08:18:38 * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 Raft Proxy > Accept: */* > < HTTP/1.1 200 OK Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD < Date: Thu, 30 May 2024 07:00:27 GMT UTC Build Time: 2024-05-27 08:24:01 < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash { [data not shown] Prometheus Prefix: tiflash_proxy_ Profile: release * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/resourcecontrol/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/resourcecontrol/tiflash/log/error.log {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00417d950 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114990210285572 MinTableBarrierTs:450114991520743428 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114990210285572, checkpointTs: 450114990210285572, state: Preparing span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114990210285572, checkpointTs: 450114990210285572, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114990210285572, checkpointTs: 450114990210285572, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318 {"id":"810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318fc84952602d 810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114990210285572,"min-table-barrier-ts":450114991520743428,"admin-job-type":0} /tidb/cdc/default/default/task/position/810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00417d950 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114990210285572 MinTableBarrierTs:450114991520743428 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114990210285572, checkpointTs: 450114990210285572, state: Preparing span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114990210285572, checkpointTs: 450114990210285572, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114990210285572, checkpointTs: 450114990210285572, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318 {"id":"810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318fc84952602d 810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114990210285572,"min-table-barrier-ts":450114991520743428,"admin-job-type":0} /tidb/cdc/default/default/task/position/810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00417d950 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114990210285572 MinTableBarrierTs:450114991520743428 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114990210285572, checkpointTs: 450114990210285572, state: Preparing span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114990210285572, checkpointTs: 450114990210285572, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114990210285572, checkpointTs: 450114990210285572, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318 {"id":"810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318fc84952602d 810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318 /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114990210285572,"min-table-barrier-ts":450114991520743428,"admin-job-type":0} /tidb/cdc/default/default/task/position/810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' [Pipeline] // container [Pipeline] } + set +x + tso='450114991311814657 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + awk -F ' ' '{print $1}' + echo 450114991311814657 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + set +x [Pipeline] // withEnv [Pipeline] // container [Pipeline] } [Thu May 30 15:00:28 CST 2024] <<<<<< START cdc server in capture_session_done_during_task case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorManagerHandleNewChangefeedDelay=sleep(2000)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_session_done_during_task.1912119123.out server --log-file /tmp/tidb_cdc_test/capture_session_done_during_task/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/capture_session_done_during_task/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] // withEnv [Pipeline] } [Pipeline] } Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [Pipeline] // node [Pipeline] } [2024/05/30 15:00:29.010 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2133269c000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3, pid:9528, start at 2024-05-30 15:00:26.680810468 +0800 CST m=+7.229832685 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:26.688 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:26.663 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:26.663 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213325280015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3, pid:9636, start at 2024-05-30 15:00:26.614504219 +0800 CST m=+5.816191529 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:26.622 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:26.620 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:26.620 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_replicate_storage_file/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_replicate_storage_file/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_file/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_file/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_file/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Pipeline] // podTemplate check diff failed 12-th time, retry later [Pipeline] // node [Pipeline] } [Pipeline] } table test.finish_mark not exists for 4-th check, retry later "is-owner": true, run task successfully [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213327940014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-cvg08, pid:3657, start at 2024-05-30 15:00:26.763700242 +0800 CST m=+5.595651039 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:26.771 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:26.773 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:26.773 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21332aa00016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-cvg08, pid:3738, start at 2024-05-30 15:00:26.95666283 +0800 CST m=+5.716459065 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:26.964 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:26.970 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:26.970 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/batch_update_to_no_batch/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/batch_update_to_no_batch/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/batch_update_to_no_batch/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/batch_update_to_no_batch/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/batch_update_to_no_batch/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Pipeline] // podTemplate [Pipeline] } [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] // withEnv [Pipeline] } [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // stage [Pipeline] } + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.cli.11081.out cli tso query --pd=http://127.0.0.1:2379 owner pid: 5501 owner id 810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318 [Thu May 30 15:00:30 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.56095611.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_hang_up_capture.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_hang_up_capture.server2 --cluster-id default --addr 127.0.0.1:8301 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 13-th time, retry later table test.finish_mark not exists for 5-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:00:31.028 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] start tidb cluster in /tmp/tidb_cdc_test/cli Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... [Thu May 30 15:00:31 CST 2024] <<<<<< START cdc server in consistent_replicate_storage_file case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_storage_file.1092510927.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 start tidb cluster in /tmp/tidb_cdc_test/processor_etcd_worker_delay Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... + set +x + tso='450114992360652801 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114992360652801 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 15:00:32 CST 2024] <<<<<< START cdc server in resourcecontrol case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.110991110111103.out server --log-file /tmp/tidb_cdc_test/resourcecontrol/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/resourcecontrol/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:00:31 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/57c4819e-04b5-4c1b-a8b6-8065f5bf5d34 {"id":"57c4819e-04b5-4c1b-a8b6-8065f5bf5d34","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c8d20e0 57c4819e-04b5-4c1b-a8b6-8065f5bf5d34 /tidb/cdc/default/default/upstream/7374683956618870037 {"id":7374683956618870037,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/57c4819e-04b5-4c1b-a8b6-8065f5bf5d34 {"id":"57c4819e-04b5-4c1b-a8b6-8065f5bf5d34","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c8d20e0 57c4819e-04b5-4c1b-a8b6-8065f5bf5d34 /tidb/cdc/default/default/upstream/7374683956618870037 {"id":7374683956618870037,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/57c4819e-04b5-4c1b-a8b6-8065f5bf5d34 {"id":"57c4819e-04b5-4c1b-a8b6-8065f5bf5d34","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c8d20e0 57c4819e-04b5-4c1b-a8b6-8065f5bf5d34 /tidb/cdc/default/default/upstream/7374683956618870037 {"id":7374683956618870037,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check diff failed 14-th time, retry later table test.finish_mark not exists for 6-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:00:33.041 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] lease 22318fc84c8d20e0 revoked [Thu May 30 15:00:33 CST 2024] <<<<<< START cdc server in batch_update_to_no_batch case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.51215123.out server --log-file /tmp/tidb_cdc_test/batch_update_to_no_batch/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/batch_update_to_no_batch/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:00:33 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003721d40 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114992831725575 MinTableBarrierTs:450114993093607431 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318 {"id":"810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/97f3646b-8fc5-4c4c-8b7a-fda932bf7f4a {"id":"97f3646b-8fc5-4c4c-8b7a-fda932bf7f4a","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318fc84952602d 810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318 /tidb/cdc/default/__cdc_meta__/owner/22318fc849526068 97f3646b-8fc5-4c4c-8b7a-fda932bf7f4a /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114992831725575,"min-table-barrier-ts":450114993093607431,"admin-job-type":0} /tidb/cdc/default/default/task/position/810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/97f3646b-8fc5-4c4c-8b7a-fda932bf7f4a/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003721d40 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114992831725575 MinTableBarrierTs:450114993093607431 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318 {"id":"810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/97f3646b-8fc5-4c4c-8b7a-fda932bf7f4a {"id":"97f3646b-8fc5-4c4c-8b7a-fda932bf7f4a","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318fc84952602d 810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318 /tidb/cdc/default/__cdc_meta__/owner/22318fc849526068 97f3646b-8fc5-4c4c-8b7a-fda932bf7f4a /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114992831725575,"min-table-barrier-ts":450114993093607431,"admin-job-type":0} /tidb/cdc/default/default/task/position/810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/97f3646b-8fc5-4c4c-8b7a-fda932bf7f4a/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003721d40 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114992831725575 MinTableBarrierTs:450114993093607431 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318 {"id":"810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/97f3646b-8fc5-4c4c-8b7a-fda932bf7f4a {"id":"97f3646b-8fc5-4c4c-8b7a-fda932bf7f4a","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318fc84952602d 810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318 /tidb/cdc/default/__cdc_meta__/owner/22318fc849526068 97f3646b-8fc5-4c4c-8b7a-fda932bf7f4a /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114992831725575,"min-table-barrier-ts":450114993093607431,"admin-job-type":0} /tidb/cdc/default/default/task/position/810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/97f3646b-8fc5-4c4c-8b7a-fda932bf7f4a/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep -v "810db3ef-e0ad-4e0b-ba0e-c07a7b3f9318" | grep id Verifying downstream PD is started... table capture_session_done_during_task.t exists check diff failed 1-th time, retry later Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:00:34 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9e76af53-e843-4321-8386-a03cd1808d63 {"id":"9e76af53-e843-4321-8386-a03cd1808d63","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c998fc4 9e76af53-e843-4321-8386-a03cd1808d63 /tidb/cdc/default/default/upstream/7374683973007108072 {"id":7374683973007108072,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9e76af53-e843-4321-8386-a03cd1808d63 {"id":"9e76af53-e843-4321-8386-a03cd1808d63","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c998fc4 9e76af53-e843-4321-8386-a03cd1808d63 /tidb/cdc/default/default/upstream/7374683973007108072 {"id":7374683973007108072,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9e76af53-e843-4321-8386-a03cd1808d63 {"id":"9e76af53-e843-4321-8386-a03cd1808d63","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c998fc4 9e76af53-e843-4321-8386-a03cd1808d63 /tidb/cdc/default/default/upstream/7374683973007108072 {"id":7374683973007108072,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ***************** properties ***************** "mysql.host"="127.0.0.1" "updateproportion"="0" "threadcount"="10" "scanproportion"="0" "insertproportion"="0" "mysql.port"="4000" "mysql.db"="consistent_replicate_storage_file" "requestdistribution"="uniform" "fieldcount"="100" "workload"="core" "dotransactions"="false" "readproportion"="0" "operationcount"="0" "mysql.user"="root" "readallfields"="true" "recordcount"="5000" ********************************************** table test.finish_mark not exists for 7-th check, retry later Verifying downstream PD is started... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/changefeed_dup_error_restart/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) "cluster-id": "default" "id": "97f3646b-8fc5-4c4c-8b7a-fda932bf7f4a", "cluster-id": "default" run task successfully [2024/05/30 15:00:35.194 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] check diff failed 15-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:00:35 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1b80435b-536b-4a5f-8b71-98c86e9bedbb {"id":"1b80435b-536b-4a5f-8b71-98c86e9bedbb","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c90d0c8 1b80435b-536b-4a5f-8b71-98c86e9bedbb /tidb/cdc/default/default/upstream/7374683964881939579 {"id":7374683964881939579,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1b80435b-536b-4a5f-8b71-98c86e9bedbb {"id":"1b80435b-536b-4a5f-8b71-98c86e9bedbb","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c90d0c8 1b80435b-536b-4a5f-8b71-98c86e9bedbb /tidb/cdc/default/default/upstream/7374683964881939579 {"id":7374683964881939579,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1b80435b-536b-4a5f-8b71-98c86e9bedbb {"id":"1b80435b-536b-4a5f-8b71-98c86e9bedbb","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c90d0c8 1b80435b-536b-4a5f-8b71-98c86e9bedbb /tidb/cdc/default/default/upstream/7374683964881939579 {"id":7374683964881939579,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.cli.11140.out cli changefeed create --start-ts=450114992360652801 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: 793ab006-9075-40a6-b943-1d3f1dbbea5b Info: {"upstream_id":7374683964881939579,"namespace":"default","id":"793ab006-9075-40a6-b943-1d3f1dbbea5b","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T15:00:36.04561729+08:00","start_ts":450114992360652801,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114992360652801,"checkpoint_ts":450114992360652801,"checkpoint_time":"2024-05-30 15:00:30.575"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... nonempty select id, val from test.availability1 where id=3 and val=3 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later check diff failed 2-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:00:36 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7f74398c-b5f2-485b-9094-3b7fd7dc5c10 {"id":"7f74398c-b5f2-485b-9094-3b7fd7dc5c10","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ca00fc6 7f74398c-b5f2-485b-9094-3b7fd7dc5c10 /tidb/cdc/default/default/upstream/7374683977349324565 {"id":7374683977349324565,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7f74398c-b5f2-485b-9094-3b7fd7dc5c10 {"id":"7f74398c-b5f2-485b-9094-3b7fd7dc5c10","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ca00fc6 7f74398c-b5f2-485b-9094-3b7fd7dc5c10 /tidb/cdc/default/default/upstream/7374683977349324565 {"id":7374683977349324565,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7f74398c-b5f2-485b-9094-3b7fd7dc5c10 {"id":"7f74398c-b5f2-485b-9094-3b7fd7dc5c10","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ca00fc6 7f74398c-b5f2-485b-9094-3b7fd7dc5c10 /tidb/cdc/default/default/upstream/7374683977349324565 {"id":7374683977349324565,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5170.out cli changefeed create '--sink-uri=mysql://root@127.0.0.1:3306/?batch-dml-enable=true' -c test ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... check diff failed 16-th time, retry later Create changefeed successfully! ID: test Info: {"upstream_id":7374683977349324565,"namespace":"default","id":"test","sink_uri":"mysql://root@127.0.0.1:3306/?batch-dml-enable=true","create_time":"2024-05-30T15:00:37.113157928+08:00","start_ts":450114994050957315,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114994050957315,"checkpoint_ts":450114994050957315,"checkpoint_time":"2024-05-30 15:00:37.023"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 8-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/changefeed_dup_error_restart Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... [2024/05/30 15:00:37.207 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] + set +x nonempty select id, val from test.availability1 where id=3 and val=3 + set +x TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 2-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 17-th time, retry later table test.finish_mark not exists for 9-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2133e3100013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-9tzt0, pid:3861, start at 2024-05-30 15:00:38.745139071 +0800 CST m=+7.751955842 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:38.753 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:38.724 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:38.724 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:00:39.220 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] [2024/05/30 15:00:39.235 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark] [ts=450114994531729413] [2024/05/30 15:00:39.235 +08:00] [INFO] [case.go:498] ["all tables synced"] [endTs=450114994531729413] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5209.out cli changefeed pause -c test check diff successfully PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... wait process cdc.test exit for 1-th time... table resourcecontrol.finish_mark not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... table test.finish_mark not exists for 10-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2133e3100013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-9tzt0, pid:3861, start at 2024-05-30 15:00:38.745139071 +0800 CST m=+7.751955842 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:38.753 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:38.724 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:38.724 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2133c3bc0005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-9tzt0, pid:3928, start at 2024-05-30 15:00:36.724103727 +0800 CST m=+5.671197580 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:36.737 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:36.719 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:36.719 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/region_merge/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/region_merge/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 18-th time, retry later wait process cdc.test exit for 3-th time... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:00:41 CST 2024] <<<<<< run test case capture_session_done_during_task success! >>>>>> + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5236.out cli changefeed update -c test '--sink-uri=mysql://root@127.0.0.1:3306/?batch-dml-enable=false' --no-confirm Diff of changefeed config: {Type:update Path:[SinkURI] From:mysql://root@127.0.0.1:3306/?batch-dml-enable=true To:mysql://root@127.0.0.1:3306/?batch-dml-enable=false} table resourcecontrol.finish_mark exists Update changefeed config successfully! ID: test Info: {"upstream_id":7374683977349324565,"namespace":"default","id":"test","sink_uri":"mysql://root@127.0.0.1:3306/?batch-dml-enable=false","create_time":"2024-05-30T15:00:37.113157928+08:00","start_ts":450114994050957315,"admin_job_type":1,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"delete_only_output_handle_key_columns":null,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"stopped","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114994483757059,"checkpoint_ts":450114994483757059,"checkpoint_time":"2024-05-30 15:00:38.674"} PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... check diff successfully nonempty select id, val from test.availability1 where id=3 and val=3 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 3-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 11-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Thu May 30 15:00:43 CST 2024] <<<<<< START cdc server in region_merge case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.region_merge.51955197.out server --log-file /tmp/tidb_cdc_test/region_merge/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/region_merge/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 1-th time... check diff failed 19-th time, retry later wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:00:43.141 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark1000] [2024/05/30 15:00:43.141 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark1000] + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5267.out cli changefeed resume -c test wait process cdc.test exit for 3-th time... PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:00:44 CST 2024] <<<<<< run test case resourcecontrol success! >>>>>> table test.finish_mark not exists for 12-th check, retry later [2024/05/30 15:00:45.142 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark1000] [2024/05/30 15:00:45.161 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark1000] [ts=450114996091486213] [2024/05/30 15:00:45.161 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark1000] [2024/05/30 15:00:45.161 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=450114996091486213] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 20-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:00:45.246 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:00:45.250 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:00:45.349 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:00:45.355 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21344304000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-p8f0f, pid:3714, start at 2024-05-30 15:00:44.897602696 +0800 CST m=+7.738850241 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:44.925 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:44.916 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:44.916 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21344304000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-p8f0f, pid:3714, start at 2024-05-30 15:00:44.897602696 +0800 CST m=+7.738850241 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:44.925 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:44.916 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:44.916 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213445f4000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-p8f0f, pid:3751, start at 2024-05-30 15:00:45.077470032 +0800 CST m=+7.767495563 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:45.093 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:45.102 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:45.102 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... + set +x TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:00:46 GMT < Content-Length: 657 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/34b2fad7-4943-46a6-abd4-df2e728f6f7c {"id":"34b2fad7-4943-46a6-abd4-df2e728f6f7c","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84cc87b82 34b2fad7-4943-46a6-abd4-df2e728f6f7c /tidb/cdc/default/default/upstream/7374684021320902295 {"id":7374684021320902295,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/34b2fad7-4943-46a6-abd4-df2e728f6f7c {"id":"34b2fad7-4943-46a6-abd4-df2e728f6f7c","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84cc87b82 34b2fad7-4943-46a6-abd4-df2e728f6f7c /tidb/cdc/default/default/upstream/7374684021320902295 {"id":7374684021320902295,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/34b2fad7-4943-46a6-abd4-df2e728f6f7c {"id":"34b2fad7-4943-46a6-abd4-df2e728f6f7c","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84cc87b82 34b2fad7-4943-46a6-abd4-df2e728f6f7c /tidb/cdc/default/default/upstream/7374684021320902295 {"id":7374684021320902295,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.region_merge.cli.5254.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ INSERT - Takes(s): 10.0, Count: 3957, OPS: 396.0, Avg(us): 11102, Min(us): 1319, Max(us): 199126, 95th(us): 89000, 99th(us): 106000 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Create changefeed successfully! ID: 00233441-b02c-4748-8a66-febee079d75d Info: {"upstream_id":7374684021320902295,"namespace":"default","id":"00233441-b02c-4748-8a66-febee079d75d","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T15:00:46.953212362+08:00","start_ts":450114996607385608,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114996607385608,"checkpoint_ts":450114996607385608,"checkpoint_time":"2024-05-30 15:00:46.775"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... check diff failed 21-th time, retry later + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5313.out cli changefeed pause -c test [2024/05/30 15:00:47.240 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark2000] [2024/05/30 15:00:47.240 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark2000] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21343f6c0004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:9629, start at 2024-05-30 15:00:44.645537943 +0800 CST m=+7.879332069 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:44.667 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:44.635 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:44.635 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21343f6c0004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:9629, start at 2024-05-30 15:00:44.645537943 +0800 CST m=+7.879332069 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:44.667 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:44.635 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:44.635 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21343dec000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:9661, start at 2024-05-30 15:00:44.572528456 +0800 CST m=+7.676913377 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:44.589 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:44.589 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:44.589 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cli/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cli/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/cli/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cli/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/cli/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table test.finish_mark not exists for 13-th check, retry later PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + set +x split_and_random_merge scale: 20 [2024/05/30 15:00:44.623 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001588780/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5336.out cli changefeed update -c test '--sink-uri=mysql://root@127.0.0.1:3306/?batch-dml-enable=true' --no-confirm [2024/05/30 15:00:49.240 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark2000] table test.finish_mark not exists for 14-th check, retry later Diff of changefeed config: {Type:update Path:[SinkURI] From:mysql://root@127.0.0.1:3306/?batch-dml-enable=false To:mysql://root@127.0.0.1:3306/?batch-dml-enable=true} nonempty select id, val from test.availability1 where id=3 and val=3 run task successfully wait process cdc.test exit for 1-th time... [2024/05/30 15:00:49.331 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark2000] [ts=450114997144780805] [2024/05/30 15:00:49.331 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark2000] [2024/05/30 15:00:49.331 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=450114997144780805] [2024/05/30 15:00:49.333 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:00:49.336 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:00:49.341 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:00:49.345 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] Update changefeed config successfully! ID: test Info: {"upstream_id":7374683977349324565,"namespace":"default","id":"test","sink_uri":"mysql://root@127.0.0.1:3306/?batch-dml-enable=true","create_time":"2024-05-30T15:00:37.113157928+08:00","start_ts":450114994050957315,"admin_job_type":1,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"delete_only_output_handle_key_columns":null,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"stopped","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114996462682133,"checkpoint_ts":450114994509971466,"checkpoint_time":"2024-05-30 15:00:38.774"} PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 22-th time, retry later [Thu May 30 15:00:50 CST 2024] <<<<<< START cdc server in processor_etcd_worker_delay case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/orchestrator/ProcessorEtcdDelay=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_etcd_worker_delay.50175019.out server --log-file /tmp/tidb_cdc_test/processor_etcd_worker_delay/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_etcd_worker_delay/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' wait process cdc.test exit for 2-th time... + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5361.out cli changefeed resume -c test cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_expire_capture [Thu May 30 15:00:51 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.58035805.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_expire_capture.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_expire_capture.server1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table test.finish_mark not exists for 15-th check, retry later [2024/05/30 15:00:51.173 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0019261e0/127.0.0.1:2379] [attempt=0] [error="rpc error: code = NotFound desc = etcdserver: requested lease not found"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213481c80012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-th7j8, pid:12721, start at 2024-05-30 15:00:48.902630967 +0800 CST m=+5.792363480 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:48.909 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:48.882 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:48.882 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213481c80012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-th7j8, pid:12721, start at 2024-05-30 15:00:48.902630967 +0800 CST m=+5.792363480 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:48.909 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:48.882 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:48.882 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213483640004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-th7j8, pid:12797, start at 2024-05-30 15:00:48.988973665 +0800 CST m=+5.811413161 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:48.996 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:48.985 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:48.985 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_dup_error_restart/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_dup_error_restart/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_dup_error_restart/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_dup_error_restart/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_dup_error_restart/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 23-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/ddl_attributes/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... [2024/05/30 15:00:51.533 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark3000] [2024/05/30 15:00:51.533 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark3000] + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/05/30 15:00:51.219 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0013541e0/127.0.0.1:2479] [attempt=0] [error="rpc error: code = NotFound desc = etcdserver: requested lease not found"] table test.finish_mark not exists for 16-th check, retry later + set +x table batch_update_to_no_batch.v1 not exists for 1-th check, retry later [2024/05/30 15:00:53.533 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark3000] [2024/05/30 15:00:53.555 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark3000] [ts=450114998162423813] [2024/05/30 15:00:53.555 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark3000] [2024/05/30 15:00:53.555 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=450114998162423813] [2024/05/30 15:00:53.558 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:00:53.561 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:00:53.566 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:00:53.627 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_dup_error_restart.cli.14160.out cli tso query --pd=http://127.0.0.1:2379 check diff successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:00:54 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003a1bcb0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114997025767431 MinTableBarrierTs:450114998336487430 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114997025767431, checkpointTs: 450114997025767431, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114997025767431, checkpointTs: 450114997025767431, state: Preparing span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114997025767431, checkpointTs: 450114997025767431, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/485a94a3-5d8f-4dbc-9d35-9e7ba6f2b6ae {"id":"485a94a3-5d8f-4dbc-9d35-9e7ba6f2b6ae","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318fc849526101 485a94a3-5d8f-4dbc-9d35-9e7ba6f2b6ae /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114997025767431,"min-table-barrier-ts":450114998336487430,"admin-job-type":0} /tidb/cdc/default/default/task/position/485a94a3-5d8f-4dbc-9d35-9e7ba6f2b6ae/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003a1bcb0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114997025767431 MinTableBarrierTs:450114998336487430 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114997025767431, checkpointTs: 450114997025767431, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114997025767431, checkpointTs: 450114997025767431, state: Preparing span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114997025767431, checkpointTs: 450114997025767431, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/485a94a3-5d8f-4dbc-9d35-9e7ba6f2b6ae {"id":"485a94a3-5d8f-4dbc-9d35-9e7ba6f2b6ae","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318fc849526101 485a94a3-5d8f-4dbc-9d35-9e7ba6f2b6ae /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114997025767431,"min-table-barrier-ts":450114998336487430,"admin-job-type":0} /tidb/cdc/default/default/task/position/485a94a3-5d8f-4dbc-9d35-9e7ba6f2b6ae/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003a1bcb0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114997025767431 MinTableBarrierTs:450114998336487430 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114997025767431, checkpointTs: 450114997025767431, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114997025767431, checkpointTs: 450114997025767431, state: Preparing span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114997025767431, checkpointTs: 450114997025767431, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/485a94a3-5d8f-4dbc-9d35-9e7ba6f2b6ae {"id":"485a94a3-5d8f-4dbc-9d35-9e7ba6f2b6ae","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318fc849526101 485a94a3-5d8f-4dbc-9d35-9e7ba6f2b6ae /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114997025767431,"min-table-barrier-ts":450114998336487430,"admin-job-type":0} /tidb/cdc/default/default/task/position/485a94a3-5d8f-4dbc-9d35-9e7ba6f2b6ae/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... table test.finish_mark not exists for 17-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/ddl_attributes Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... table batch_update_to_no_batch.v1 exists table batch_update_to_no_batch.recover_and_insert not exists for 1-th check, retry later *************************** 1. row *************************** count(distinct region_id): 1 Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:00:54 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c76b8a2c-32b1-4328-b1c3-59da250f045e {"id":"c76b8a2c-32b1-4328-b1c3-59da250f045e","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84cdc5acb c76b8a2c-32b1-4328-b1c3-59da250f045e /tidb/cdc/default/default/upstream/7374684048398385940 {"id":7374684048398385940,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c76b8a2c-32b1-4328-b1c3-59da250f045e {"id":"c76b8a2c-32b1-4328-b1c3-59da250f045e","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84cdc5acb c76b8a2c-32b1-4328-b1c3-59da250f045e /tidb/cdc/default/default/upstream/7374684048398385940 {"id":7374684048398385940,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c76b8a2c-32b1-4328-b1c3-59da250f045e {"id":"c76b8a2c-32b1-4328-b1c3-59da250f045e","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84cdc5acb c76b8a2c-32b1-4328-b1c3-59da250f045e /tidb/cdc/default/default/upstream/7374684048398385940 {"id":7374684048398385940,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x INSERT - Takes(s): 20.0, Count: 4087, OPS: 204.4, Avg(us): 40969, Min(us): 1319, Max(us): 5757943, 95th(us): 91000, 99th(us): 200000 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.10899.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='450114998457073668 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114998457073668 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x wait process cdc.test exit for 3-th time... "is-owner": true, run task successfully [2024/05/30 15:00:55.643 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark4000] [2024/05/30 15:00:55.643 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark4000] cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:00:55 CST 2024] <<<<<< run test case clustered_index success! >>>>>> ***************** properties ***************** "recordcount"="10" "scanproportion"="0" "updateproportion"="0" "mysql.port"="4000" "readproportion"="0" "insertproportion"="0" "threadcount"="2" "requestdistribution"="uniform" "operationcount"="0" "readallfields"="true" "workload"="core" "mysql.user"="root" "dotransactions"="false" "mysql.db"="changefeed_dup_error_restart" "mysql.host"="127.0.0.1" ********************************************** Run finished, takes 10.739571ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1929.3, Avg(us): 2045, Min(us): 1012, Max(us): 5471, 95th(us): 6000, 99th(us): 6000 [Thu May 30 15:00:56 CST 2024] <<<<<< START cdc server in changefeed_dup_error_restart case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLDuplicateEntryError=5%return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_dup_error_restart.1420914211.out server --log-file /tmp/tidb_cdc_test/changefeed_dup_error_restart/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_dup_error_restart/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x + tso='450114998694313985 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114998694313985 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x owner pid: 5808 owner id 485a94a3-5d8f-4dbc-9d35-9e7ba6f2b6ae process status: T ETCDCTL_API=3 etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix | grep -v '485a94a3-5d8f-4dbc-9d35-9e7ba6f2b6ae' /tidb/cdc/default/__cdc_meta__/owner/22318fc849526101 run task successfully process status: S nonempty select id, val from test.availability1 where id=2 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later [2024/05/30 15:00:57.313 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0016ca1e0/127.0.0.1:2379] [attempt=0] [error="rpc error: code = NotFound desc = etcdserver: requested lease not found"] table test.finish_mark not exists for 18-th check, retry later [2024/05/30 15:00:51.485 +08:00] [WARN] [diff.go:179] ["table struct is not equal"] [reason="column num not equal, one is 2 another is 6"] table batch_update_to_no_batch.recover_and_insert not exists for 2-th check, retry later [2024/05/30 15:00:57.643 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark4000] [Thu May 30 15:00:57 CST 2024] <<<<<< START cdc server in cli case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.1092810930.out server --log-file /tmp/tidb_cdc_test/cli/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cli/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/05/30 15:00:57.029 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0019263c0/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] table test.finish_mark not exists for 19-th check, retry later split_and_random_merge scale: 40 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [2024/05/30 15:00:58.554 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001588780/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] table batch_update_to_no_batch.recover_and_insert not exists for 3-th check, retry later [2024/05/30 15:00:59.445 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark5000] [2024/05/30 15:00:59.445 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark5000] nonempty select id, val from test.availability1 where id=2 and val=22 run task successfully wait process cdc.test exit for 1-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:00:59 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b6eb86fb-2bcb-493e-941c-831aa941b058 {"id":"b6eb86fb-2bcb-493e-941c-831aa941b058","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84cf409e0 b6eb86fb-2bcb-493e-941c-831aa941b058 /tidb/cdc/default/default/upstream/7374684077513943424 {"id":7374684077513943424,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b6eb86fb-2bcb-493e-941c-831aa941b058 {"id":"b6eb86fb-2bcb-493e-941c-831aa941b058","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84cf409e0 b6eb86fb-2bcb-493e-941c-831aa941b058 /tidb/cdc/default/default/upstream/7374684077513943424 {"id":7374684077513943424,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b6eb86fb-2bcb-493e-941c-831aa941b058 {"id":"b6eb86fb-2bcb-493e-941c-831aa941b058","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84cf409e0 b6eb86fb-2bcb-493e-941c-831aa941b058 /tidb/cdc/default/default/upstream/7374684077513943424 {"id":7374684077513943424,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_dup_error_restart.cli.14254.out cli changefeed create --start-ts=450114998457073668 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } [2024/05/30 15:00:59.657 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark4000] [2024/05/30 15:00:59.766 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark4000] [ts=450114999530291205] [2024/05/30 15:00:59.767 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark4000] [2024/05/30 15:00:59.767 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=450114999530291205] [2024/05/30 15:00:59.835 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:00:59.841 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:00:59.846 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:00:59.852 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc already exists) Create changefeed successfully! ID: 0eec6602-ce1e-4d27-9522-dc787c867582 Info: {"upstream_id":7374684077513943424,"namespace":"default","id":"0eec6602-ce1e-4d27-9522-dc787c867582","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T15:00:59.950830214+08:00","start_ts":450114998457073668,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114998457073668,"checkpoint_ts":450114998457073668,"checkpoint_time":"2024-05-30 15:00:53.831"} PASS [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } wait process cdc.test exit for 2-th time... coverage: 2.4% of statements in github.com/pingcap/tiflow/... [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // container [Pipeline] } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:01:00 GMT < Content-Length: 657 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5e5c9759-43e6-46b2-be4d-29c8a081864b {"id":"5e5c9759-43e6-46b2-be4d-29c8a081864b","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84cdc6b57 5e5c9759-43e6-46b2-be4d-29c8a081864b /tidb/cdc/default/default/upstream/7374684044726023853 {"id":7374684044726023853,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5e5c9759-43e6-46b2-be4d-29c8a081864b {"id":"5e5c9759-43e6-46b2-be4d-29c8a081864b","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84cdc6b57 5e5c9759-43e6-46b2-be4d-29c8a081864b /tidb/cdc/default/default/upstream/7374684044726023853 {"id":7374684044726023853,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5e5c9759-43e6-46b2-be4d-29c8a081864b {"id":"5e5c9759-43e6-46b2-be4d-29c8a081864b","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84cdc6b57 5e5c9759-43e6-46b2-be4d-29c8a081864b /tidb/cdc/default/default/upstream/7374684044726023853 {"id":7374684044726023853,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.10964.out cli changefeed create --start-ts=450114998694313985 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --tz=Asia/Shanghai -c=custom-changefeed-name [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } table test.finish_mark not exists for 20-th check, retry later [WARN] --tz is deprecated in changefeed settings. [Pipeline] // podTemplate table batch_update_to_no_batch.recover_and_insert exists table batch_update_to_no_batch.finish_mark not exists for 1-th check, retry later [Pipeline] } Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7374684044726023853,"namespace":"default","id":"custom-changefeed-name","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T15:01:01.465027464+08:00","start_ts":450114998694313985,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114998694313985,"checkpoint_ts":450114998694313985,"checkpoint_time":"2024-05-30 15:00:54.736"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x [Pipeline] // withEnv [Pipeline] } [2024/05/30 15:01:01.446 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark5000] [Pipeline] // stage [Pipeline] } cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit sql_check run sql_check, 127.0.0.1 run task successfully run test case test_stop_processor [Thu May 30 15:01:02 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.59905992.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_stop_processor.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_stop_processor --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table test.finish_mark not exists for 21-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table batch_update_to_no_batch.finish_mark not exists for 2-th check, retry later + set +x table test.simple not exists for 1-th check, retry later [2024/05/30 15:01:03.458 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark5000] [2024/05/30 15:01:03.462 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0013541e0/127.0.0.1:2479] [attempt=0] [error="rpc error: code = NotFound desc = etcdserver: requested lease not found"] INSERT - Takes(s): 30.0, Count: 4376, OPS: 145.9, Avg(us): 68093, Min(us): 1319, Max(us): 5757943, 95th(us): 93000, 99th(us): 0 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table batch_update_to_no_batch.finish_mark not exists for 3-th check, retry later table processor_delay.t1 exists table processor_delay.t2 exists table processor_delay.t3 not exists for 1-th check, retry later [2024/05/30 15:01:05.476 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark5000] [2024/05/30 15:01:05.491 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark5000] [ts=450115001347473411] [2024/05/30 15:01:05.491 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark5000] [2024/05/30 15:01:05.491 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=450115001347473411] table test.simple exists table test.`simple-dash` not exists for 1-th check, retry later table test.finish_mark not exists for 22-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:01:05 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0035682d0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114999660314625 MinTableBarrierTs:450115001233440771 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114999660314625, checkpointTs: 450114999660314625, state: Preparing span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114999660314625, checkpointTs: 450114999660314625, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114999660314625, checkpointTs: 450114999660314625, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/39d019e3-f01f-4b0b-ae5f-b0e4e7e1eb9d {"id":"39d019e3-f01f-4b0b-ae5f-b0e4e7e1eb9d","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318fc84952616e 39d019e3-f01f-4b0b-ae5f-b0e4e7e1eb9d /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114999660314625,"min-table-barrier-ts":450115001233440771,"admin-job-type":0} /tidb/cdc/default/default/task/position/39d019e3-f01f-4b0b-ae5f-b0e4e7e1eb9d/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0035682d0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114999660314625 MinTableBarrierTs:450115001233440771 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114999660314625, checkpointTs: 450114999660314625, state: Preparing span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114999660314625, checkpointTs: 450114999660314625, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114999660314625, checkpointTs: 450114999660314625, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/39d019e3-f01f-4b0b-ae5f-b0e4e7e1eb9d {"id":"39d019e3-f01f-4b0b-ae5f-b0e4e7e1eb9d","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318fc84952616e 39d019e3-f01f-4b0b-ae5f-b0e4e7e1eb9d /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114999660314625,"min-table-barrier-ts":450115001233440771,"admin-job-type":0} /tidb/cdc/default/default/task/position/39d019e3-f01f-4b0b-ae5f-b0e4e7e1eb9d/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/+ grep -q 'failed to get info:' default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/56d8a316-8373-4096-9cb8-564fb0ca6f6b {UpstreamID:7374683055714277379 Namespace:default ID:56d8a316-8373-4096-9cb8-564fb0ca6f6b SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 14:57:03.610004086 +0800 CST StartTs:450114936732647425 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0035682d0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114938095796234} {CheckpointTs:450114999660314625 MinTableBarrierTs:450115001233440771 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114999660314625, checkpointTs: 450114999660314625, state: Preparing span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 450114999660314625, checkpointTs: 450114999660314625, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114999660314625, checkpointTs: 450114999660314625, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/39d019e3-f01f-4b0b-ae5f-b0e4e7e1eb9d {"id":"39d019e3-f01f-4b0b-ae5f-b0e4e7e1eb9d","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318fc84952616e 39d019e3-f01f-4b0b-ae5f-b0e4e7e1eb9d /tidb/cdc/default/default/changefeed/info/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"upstream-id":7374683055714277379,"namespace":"default","changefeed-id":"56d8a316-8373-4096-9cb8-564fb0ca6f6b","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T14:57:03.610004086+08:00","start-ts":450114936732647425,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114938095796234} /tidb/cdc/default/default/changefeed/status/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":450114999660314625,"min-table-barrier-ts":450115001233440771,"admin-job-type":0} /tidb/cdc/default/default/task/position/39d019e3-f01f-4b0b-ae5f-b0e4e7e1eb9d/56d8a316-8373-4096-9cb8-564fb0ca6f6b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683055714277379 {"id":7374683055714277379,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' [2024/05/30 15:01:05.972 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:01:05.976 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:01:05.990 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:01:06.002 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] *************************** 1. row *************************** count(distinct region_id): 1 "is-owner": true, run task successfully table batch_update_to_no_batch.finish_mark not exists for 4-th check, retry later table processor_delay.t3 exists table processor_delay.t4 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.`simple-dash` exists + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s table test.finish_mark not exists for 23-th check, retry later + info='{ "upstream_id": 7374684044726023853, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 450114999310090248, "checkpoint_time": "2024-05-30 15:00:57.085", "error": null }' + echo '{ "upstream_id": 7374684044726023853, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 450114999310090248, "checkpoint_time": "2024-05-30 15:00:57.085", "error": null }' { "upstream_id": 7374684044726023853, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 450114999310090248, "checkpoint_time": "2024-05-30 15:00:57.085", "error": null } ++ echo '{' '"upstream_id":' 7374684044726023853, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 450114999310090248, '"checkpoint_time":' '"2024-05-30' '15:00:57.085",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374684044726023853, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 450114999310090248, '"checkpoint_time":' '"2024-05-30' '15:00:57.085",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] changefeed count 1 check pass, pd_addr: http://127.0.0.1:2379 changefeed count 1 check pass, pd_addr: http://127.0.0.1:2679 changefeed count 1 check pass, pd_addr: http://127.0.0.1:2779 owner pid: 5995 owner id 39d019e3-f01f-4b0b-ae5f-b0e4e7e1eb9d changefeed count 1 check pass, pd_addr: http://127.0.0.1:2379,http://127.0.0.1:2679,http://127.0.0.1:2779 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213592c8000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-lpxxn, pid:20205, start at 2024-05-30 15:01:06.366991676 +0800 CST m=+5.606775356 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:03:06.377 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:01:06.354 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:51:06.354 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213592c8000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-lpxxn, pid:20205, start at 2024-05-30 15:01:06.366991676 +0800 CST m=+5.606775356 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:03:06.377 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:01:06.354 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:51:06.354 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21359440000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-lpxxn, pid:20282, start at 2024-05-30 15:01:06.460517612 +0800 CST m=+5.636449295 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:03:06.467 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:01:06.448 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:51:06.448 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_attributes/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_attributes/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table processor_delay.t4 exists table processor_delay.t5 not exists for 1-th check, retry later Error: [CDC:ErrChangefeedUpdateRefused]changefeed update error: can only update changefeed config when it is stopped or failed update changefeed config should fail when changefeed is running, got Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11241.out cli changefeed --changefeed-id custom-changefeed-name pause table test.finish_mark not exists for 24-th check, retry later PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... table batch_update_to_no_batch.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... changefeed id: 56d8a316-8373-4096-9cb8-564fb0ca6f6b curl -s -d "cf-id=56d8a316-8373-4096-9cb8-564fb0ca6f6b&admin-job=1" http://127.0.0.1:8300/capture/owner/admin | grep true "status": true, run task successfully % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 89 100 35 100 54 790 1219 --:--:-- --:--:-- --:--:-- 1227 { "status": true, "message": "" }nonempty select id, val from test.availability1 where id=4 and val=4 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later [2024/05/30 15:01:09.639 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark6000] [2024/05/30 15:01:09.639 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark6000] split_and_random_merge scale: 80 wait process cdc.test exit for 2-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/processor_resolved_ts_fallback/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + set +x table processor_delay.t5 exists table processor_delay.t6 not exists for 1-th check, retry later wait process cdc.test exit for 3-th time... Run finished, takes 35.974548092s INSERT - Takes(s): 36.0, Count: 5000, OPS: 139.0, Avg(us): 70639, Min(us): 1319, Max(us): 5757943, 95th(us): 107000, 99th(us): 907000 table test.finish_mark not exists for 25-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.cli.21652.out cli tso query --pd=http://127.0.0.1:2379 cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:01:11 CST 2024] <<<<<< run test case batch_update_to_no_batch success! >>>>>> [2024/05/30 15:01:11.639 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark6000] [2024/05/30 15:01:11.660 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark6000] [ts=450115002911948805] [2024/05/30 15:01:11.660 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark6000] [2024/05/30 15:01:11.660 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=450115002911948805] [2024/05/30 15:01:11.662 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:01:11.728 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:01:11.732 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:01:11.737 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] nonempty select id, val from test.availability1 where id=4 and val=4 run task successfully test_stop_processor pass wait process cdc.test exit for 1-th time... + set +x + tso='450115003089682439 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115003089682439 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 15:01:12 CST 2024] <<<<<< START cdc server in ddl_attributes case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.2168521687.out server --log-file /tmp/tidb_cdc_test/ddl_attributes/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_attributes/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 2-th time... table test.finish_mark not exists for 26-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 15:01:13 CST 2024] <<<<<< run test case availability success! >>>>>> table processor_delay.t6 exists table processor_delay.t7 exists table processor_delay.t8 not exists for 1-th check, retry later [2024/05/30 15:01:13.643 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark7000] [2024/05/30 15:01:13.643 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark7000] start tidb cluster in /tmp/tidb_cdc_test/processor_resolved_ts_fallback Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=stopped + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7374684044726023853, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 450115001983959041, "checkpoint_time": "2024-05-30 15:01:07.285", "error": null }' + echo '{ "upstream_id": 7374684044726023853, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 450115001983959041, "checkpoint_time": "2024-05-30 15:01:07.285", "error": null }' { "upstream_id": 7374684044726023853, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 450115001983959041, "checkpoint_time": "2024-05-30 15:01:07.285", "error": null } ++ echo '{' '"upstream_id":' 7374684044726023853, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 450115001983959041, '"checkpoint_time":' '"2024-05-30' '15:01:07.285",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7374684044726023853, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 450115001983959041, '"checkpoint_time":' '"2024-05-30' '15:01:07.285",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11315.out cli changefeed update --pd=http://127.0.0.1:2379,http://127.0.0.1:2679,http://127.0.0.1:2779 --config=/tmp/tidb_cdc_test/cli/changefeed.toml --no-confirm --changefeed-id custom-changefeed-name Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} Update changefeed config successfully! ID: custom-changefeed-name Info: {"upstream_id":7374684044726023853,"namespace":"default","id":"custom-changefeed-name","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T15:01:01.465027464+08:00","start_ts":450114998694313985,"admin_job_type":1,"config":{"memory_quota":1073741824,"case_sensitive":true,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"delete_only_output_handle_key_columns":null,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"stopped","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115002508247042,"checkpoint_ts":450115001983959041,"checkpoint_time":"2024-05-30 15:01:07.285"} PASS coverage: 2.8% of statements in github.com/pingcap/tiflow/... table test.finish_mark not exists for 27-th check, retry later table processor_delay.t8 exists table processor_delay.t9 not exists for 1-th check, retry later Verifying downstream PD is started... [2024/05/30 15:01:15.643 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark7000] + set +x + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:01:16 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/08ca5f3e-72bb-4eb7-a579-c3f23c620604 {"id":"08ca5f3e-72bb-4eb7-a579-c3f23c620604","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84d38edc3 08ca5f3e-72bb-4eb7-a579-c3f23c620604 /tidb/cdc/default/default/upstream/7374684148755578732 {"id":7374684148755578732,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/08ca5f3e-72bb-4eb7-a579-c3f23c620604 {"id":"08ca5f3e-72bb-4eb7-a579-c3f23c620604","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84d38edc3 08ca5f3e-72bb-4eb7-a579-c3f23c620604 /tidb/cdc/default/default/upstream/7374684148755578732 {"id":7374684148755578732,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/08ca5f3e-72bb-4eb7-a579-c3f23c620604 {"id":"08ca5f3e-72bb-4eb7-a579-c3f23c620604","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84d38edc3 08ca5f3e-72bb-4eb7-a579-c3f23c620604 /tidb/cdc/default/default/upstream/7374684148755578732 {"id":7374684148755578732,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.cli.21734.out cli changefeed create --start-ts=450115003089682439 --sink-uri=mysql://root@127.0.0.1:3306/ + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11360.out cli changefeed --changefeed-id custom-changefeed-name resume Create changefeed successfully! ID: a8372214-bfc7-4bec-a56d-618611e7ed30 Info: {"upstream_id":7374684148755578732,"namespace":"default","id":"a8372214-bfc7-4bec-a56d-618611e7ed30","sink_uri":"mysql://root@127.0.0.1:3306/","create_time":"2024-05-30T15:01:16.493737365+08:00","start_ts":450115003089682439,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115003089682439,"checkpoint_ts":450115003089682439,"checkpoint_time":"2024-05-30 15:01:11.503"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... *************************** 1. row *************************** count(distinct region_id): 3 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... table processor_delay.t9 exists table processor_delay.t10 not exists for 1-th check, retry later Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table test.finish_mark not exists for 28-th check, retry later + set +x [2024/05/30 15:01:17.636 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark8000] [2024/05/30 15:01:17.636 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark8000] [2024/05/30 15:01:17.656 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark7000] [2024/05/30 15:01:17.746 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark7000] [ts=450115004244164613] [2024/05/30 15:01:17.746 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark7000] [2024/05/30 15:01:17.746 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=450115004244164613] [2024/05/30 15:01:17.749 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:01:17.754 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:01:17.828 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:01:17.835 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] + set +x table processor_delay.t10 not exists for 2-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... table test.finish_mark not exists for 29-th check, retry later Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:01:19.636 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark8000] [2024/05/30 15:01:19.652 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark8000] [ts=450115005140172809] [2024/05/30 15:01:19.653 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark8000] [2024/05/30 15:01:19.653 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=450115005140172809] [2024/05/30 15:01:19.657 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:01:19.729 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:01:19.735 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:01:19.741 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] table processor_delay.t10 exists table processor_delay.t11 exists table processor_delay.t12 not exists for 1-th check, retry later table test.finish_mark not exists for 30-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7374684044726023853, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 450115005391831042, "checkpoint_time": "2024-05-30 15:01:20.285", "error": null }' + echo '{ "upstream_id": 7374684044726023853, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 450115005391831042, "checkpoint_time": "2024-05-30 15:01:20.285", "error": null }' { "upstream_id": 7374684044726023853, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 450115005391831042, "checkpoint_time": "2024-05-30 15:01:20.285", "error": null } ++ echo '{' '"upstream_id":' 7374684044726023853, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 450115005391831042, '"checkpoint_time":' '"2024-05-30' '15:01:20.285",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374684044726023853, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 450115005391831042, '"checkpoint_time":' '"2024-05-30' '15:01:20.285",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11448.out cli changefeed --changefeed-id custom-changefeed-name remove [2024/05/30 15:01:20.521 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001588780/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:01:20.714 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0019263c0/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:01:19.580 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0016cad20/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:01:19.648 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0016e23c0/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:01:20.519 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0016cad20/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:01:20.520 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0016e23c0/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] table region_merge.t1 exists check diff failed 1-th time, retry later [2024/05/30 15:01:21.895 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0016ca1e0/127.0.0.1:2379] [attempt=0] [error="rpc error: code = NotFound desc = etcdserver: requested lease not found"] {"level":"warn","ts":"2024-05-30T15:01:22.316231+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0006f3180/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Changefeed remove successfully. ID: custom-changefeed-name CheckpointTs: 450115005653975042 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/ PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... [2024/05/30 15:01:22.339 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark9000] [2024/05/30 15:01:22.339 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark9000] [2024/05/30 15:01:22.897 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0013541e0/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:01:22.999 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0016e23c0/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:01:22.897 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0019261e0/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:01:23.018 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001588780/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:01:23.225 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0019263c0/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] table test.finish_mark not exists for 31-th check, retry later table processor_delay.t12 exists table processor_delay.t13 not exists for 1-th check, retry later check diff successfully + set +x ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... [2024/05/30 15:01:24.339 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark9000] [2024/05/30 15:01:24.446 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark9000] [ts=450115006346035205] [2024/05/30 15:01:24.446 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark9000] [2024/05/30 15:01:24.446 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=450115006346035205] [2024/05/30 15:01:24.449 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:01:24.527 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:01:24.532 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:01:24.538 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] wait process cdc.test exit for 2-th time... table test.finish_mark not exists for 32-th check, retry later wait process cdc.test exit for 3-th time... table processor_delay.t13 exists table processor_delay.t14 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/http_proxies/run.sh using Sink-Type: mysql... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/http_proxies/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/http_proxies ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/http_proxies + source /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/http_proxies/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/http_proxies + CDC_BINARY=cdc.test + SINK_TYPE=mysql + TEST_HOST_LIST=(UP_TIDB_HOST UP_PD_HOST_{1..3} UP_TIKV_HOST_{1..3}) + lan_addrs=($(hostname -I)) ++ hostname -I + lan_addr=10.233.106.86 + export UP_TIDB_HOST=10.233.106.86 UP_PD_HOST_1=10.233.106.86 UP_PD_HOST_2=10.233.106.86 UP_PD_HOST_3=10.233.106.86 UP_TIKV_HOST_1=10.233.106.86 UP_TIKV_HOST_2=10.233.106.86 UP_TIKV_HOST_3=10.233.106.86 + UP_TIDB_HOST=10.233.106.86 + UP_PD_HOST_1=10.233.106.86 + UP_PD_HOST_2=10.233.106.86 + UP_PD_HOST_3=10.233.106.86 + UP_TIKV_HOST_1=10.233.106.86 + UP_TIKV_HOST_2=10.233.106.86 + UP_TIKV_HOST_3=10.233.106.86 + proxy_pid= ++ shuf -i 10081-20081 -n1 + proxy_port=12770 + trap 'stop_tidb_cluster && stop_proxy' EXIT + '[' mysql == mysql ']' + prepare + for host in '"${TEST_HOST_LIST[@]}"' ++ printenv UP_TIDB_HOST + echo 'UP_TIDB_HOST 10.233.106.86' UP_TIDB_HOST 10.233.106.86 + case $(printenv $host) in ++ printenv UP_TIDB_HOST + for host in '"${TEST_HOST_LIST[@]}"' ++ printenv UP_PD_HOST_1 + echo 'UP_PD_HOST_1 10.233.106.86' UP_PD_HOST_1 10.233.106.86 + case $(printenv $host) in ++ printenv UP_PD_HOST_1 + for host in '"${TEST_HOST_LIST[@]}"' ++ printenv UP_PD_HOST_2 + echo 'UP_PD_HOST_2 10.233.106.86' UP_PD_HOST_2 10.233.106.86 + case $(printenv $host) in ++ printenv UP_PD_HOST_2 + for host in '"${TEST_HOST_LIST[@]}"' ++ printenv UP_PD_HOST_3 + echo 'UP_PD_HOST_3 10.233.106.86' UP_PD_HOST_3 10.233.106.86 + case $(printenv $host) in ++ printenv UP_PD_HOST_3 + for host in '"${TEST_HOST_LIST[@]}"' ++ printenv UP_TIKV_HOST_1 + echo 'UP_TIKV_HOST_1 10.233.106.86' UP_TIKV_HOST_1 10.233.106.86 + case $(printenv $host) in ++ printenv UP_TIKV_HOST_1 + for host in '"${TEST_HOST_LIST[@]}"' ++ printenv UP_TIKV_HOST_2 + echo 'UP_TIKV_HOST_2 10.233.106.86' UP_TIKV_HOST_2 10.233.106.86 + case $(printenv $host) in ++ printenv UP_TIKV_HOST_2 + for host in '"${TEST_HOST_LIST[@]}"' ++ printenv UP_TIKV_HOST_3 + echo 'UP_TIKV_HOST_3 10.233.106.86' UP_TIKV_HOST_3 10.233.106.86 + case $(printenv $host) in ++ printenv UP_TIKV_HOST_3 + rm -rf /tmp/tidb_cdc_test/http_proxies + mkdir -p /tmp/tidb_cdc_test/http_proxies + stop_tidb_cluster [2024/05/30 15:01:25.367 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001926000/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:01:25.431 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001926000/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:01:25.495 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001588780/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:01:25.703 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0019263c0/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2136b0640013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4vpcx, pid:4517, start at 2024-05-30 15:01:24.669381387 +0800 CST m=+5.352356484 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:03:24.678 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:01:24.682 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:51:24.682 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:01:25 CST 2024] <<<<<< run test case region_merge success! >>>>>> [2024/05/30 15:01:26.161 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc002114000/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:01:26.161 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001a2e000/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/changefeed_resume_with_checkpoint_ts/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2024/05/30 15:01:26.098 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0016e23c0/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:01:26.418 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0016e2000/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:01:26.418 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0019a2000/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:01:26.546 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark10000] [2024/05/30 15:01:26.546 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark10000] changefeed count 0 check pass, pd_addr: http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11508.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --tz=Asia/Shanghai -c=custom-changefeed-name [WARN] --tz is deprecated in changefeed settings. table processor_delay.t14 exists table processor_delay.t15 not exists for 1-th check, retry later + start_tidb_cluster --workdir /tmp/tidb_cdc_test/http_proxies The 1 times to try to start tidb cluster... Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7374684044726023853,"namespace":"default","id":"custom-changefeed-name","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T15:01:27.190970666+08:00","start_ts":450115007174410242,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115007174410242,"checkpoint_ts":450115007174410242,"checkpoint_time":"2024-05-30 15:01:27.085"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table test.finish_mark not exists for 33-th check, retry later [2024/05/30 15:01:27.432 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001926000/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2136b0640013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4vpcx, pid:4517, start at 2024-05-30 15:01:24.669381387 +0800 CST m=+5.352356484 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:03:24.678 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:01:24.682 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:51:24.682 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2136b1f40014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-4vpcx, pid:4591, start at 2024-05-30 15:01:24.755971524 +0800 CST m=+5.381122544 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:03:24.763 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:01:24.733 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:51:24.733 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } pass check, checkpoint tso not forward after 10s run task successfully wait process 7863 exit for 1-th time... wait process 7863 exit for 2-th time... wait process 7863 exit for 3-th time... wait process 7863 exit for 4-th time... wait process 7863 exit for 5-th time... /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (7863) - No such process wait process 7863 exit for 6-th time... process 7863 already exit [Thu May 30 15:01:22 CST 2024] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.83578359.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:01:25 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7374683728060867366 Namespace:default ID:ddl-only-block-related-table SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2024-05-30 15:00:01.975774647 +0800 CST StartTs:450114984828993541 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003b7e2d0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114984854945799} {CheckpointTs:450114989115047947 MinTableBarrierTs:450115006442766342 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114989115047947, checkpointTs: 450114989115047947, state: Preparing span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 450114989115047947, checkpointTs: 450114989115047947, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114989115047947, checkpointTs: 450114989115047947, state: Preparing span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 450114989115047947, checkpointTs: 450114989115047947, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ccff95bd-8919-40f4-93cd-78af41be93f3 {"id":"ccff95bd-8919-40f4-93cd-78af41be93f3","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bbf5d90 ccff95bd-8919-40f4-93cd-78af41be93f3 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7374683728060867366,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2024-05-30T15:00:01.975774647+08:00","start-ts":450114984828993541,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114984854945799} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":450114989115047947,"min-table-barrier-ts":450115006547623938,"admin-job-type":0} /tidb/cdc/default/default/task/position/ccff95bd-8919-40f4-93cd-78af41be93f3/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683728060867366 {"id":7374683728060867366,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7374683728060867366 Namespace:default ID:ddl-only-block-related-table SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2024-05-30 15:00:01.975774647 +0800 CST StartTs:450114984828993541 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003b7e2d0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114984854945799} {CheckpointTs:450114989115047947 MinTableBarrierTs:450115006442766342 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114989115047947, checkpointTs: 450114989115047947, state: Preparing span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 450114989115047947, checkpointTs: 450114989115047947, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114989115047947, checkpointTs: 450114989115047947, state: Preparing span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 450114989115047947, checkpointTs: 450114989115047947, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ccff95bd-8919-40f4-93cd-78af41be93f3 {"id":"ccff95bd-8919-40f4-93cd-78af41be93f3","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bbf5d90 ccff95bd-8919-40f4-93cd-78af41be93f3 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7374683728060867366,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2024-05-30T15:00:01.975774647+08:00","start-ts":450114984828993541,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114984854945799} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":450114989115047947,"min-table-barrier-ts":450115006547623938,"admin-job-type":0} /tidb/cdc/default/default/task/position/ccff95bd-8919-40f4-93cd-78af41be93f3/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683728060867366 {"id":7374683728060867366,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7374683728060867366 Namespace:default ID:ddl-only-block-related-table SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2024-05-30 15:00:01.975774647 +0800 CST StartTs:450114984828993541 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003b7e2d0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114984854945799} {CheckpointTs:450114989115047947 MinTableBarrierTs:450115006442766342 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450114989115047947, checkpointTs: 450114989115047947, state: Preparing span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 450114989115047947, checkpointTs: 450114989115047947, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114989115047947, checkpointTs: 450114989115047947, state: Preparing span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 450114989115047947, checkpointTs: 450114989115047947, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ccff95bd-8919-40f4-93cd-78af41be93f3 {"id":"ccff95bd-8919-40f4-93cd-78af41be93f3","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84bbf5d90 ccff95bd-8919-40f4-93cd-78af41be93f3 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7374683728060867366,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2024-05-30T15:00:01.975774647+08:00","start-ts":450114984828993541,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114984854945799} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":450114989115047947,"min-table-barrier-ts":450115006547623938,"admin-job-type":0} /tidb/cdc/default/default/task/position/ccff95bd-8919-40f4-93cd-78af41be93f3/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683728060867366 {"id":7374683728060867366,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check diff successfully check_ts_forward ddl-only-block-related-table changefeed is working normally rts: 450114989115047940->450115006967054340 checkpoint: 450114989115047940->450115006967054340 run task successfully wait process cdc.test exit for 1-th time... + set +x wait process cdc.test exit for 2-th time... start tidb cluster in /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... [2024/05/30 15:01:28.625 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark10000] [2024/05/30 15:01:28.644 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark10000] [ts=450115007420825605] [2024/05/30 15:01:28.644 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark10000] [2024/05/30 15:01:28.644 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=450115007420825605] [2024/05/30 15:01:28.647 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:01:28.728 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:01:28.736 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:01:28.742 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] table processor_delay.t15 exists table processor_delay.t16 not exists for 1-th check, retry later wait process cdc.test exit for 3-th time... table test.finish_mark not exists for 34-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/http_proxies Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:01:29 CST 2024] <<<<<< run test case ddl_only_block_related_table success! >>>>>> table big_txn.usertable1 exists [Thu May 30 15:01:29 CST 2024] <<<<<< START cdc server in processor_resolved_ts_fallback case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkExecDMLError=return(true);github.com/pingcap/tiflow/cdc/sink/dmlsink/mq/dmlproducer/KafkaSinkAsyncSendError=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.58585860.out server --log-file /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Verifying downstream PD is started... [2024/05/30 15:01:30.737 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark11000] [2024/05/30 15:01:30.737 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark11000] table test.finish_mark not exists for 35-th check, retry later pass test: get status pass test: health pass test: changefeed apis pass test: delete changefeed apis pass test: capture apis pass test: processor apis pass test: owner apis pass test: set log level + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit ++ date + echo '[Thu May 30 15:01:29 CST 2024] <<<<<< run test case api_v2 success! >>>>>>' [Thu May 30 15:01:29 CST 2024] <<<<<< run test case api_v2 success! >>>>>> + stop_tidb_cluster table processor_delay.t16 exists table processor_delay.t17 not exists for 1-th check, retry later + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7374684044726023853, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 450115008275415043, "checkpoint_time": "2024-05-30 15:01:31.285", "error": null }' + echo '{ "upstream_id": 7374684044726023853, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 450115008275415043, "checkpoint_time": "2024-05-30 15:01:31.285", "error": null }' { "upstream_id": 7374684044726023853, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 450115008275415043, "checkpoint_time": "2024-05-30 15:01:31.285", "error": null } ++ echo '{' '"upstream_id":' 7374684044726023853, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 450115008275415043, '"checkpoint_time":' '"2024-05-30' '15:01:31.285",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374684044726023853, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 450115008275415043, '"checkpoint_time":' '"2024-05-30' '15:01:31.285",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] [2024/05/30 15:01:32.737 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark11000] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11579.out cli unsafe delete-service-gc-safepoint table test.finish_mark not exists for 36-th check, retry later Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:01:32 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/81a5b110-bd87-4bb1-b17b-ff110c4fe0b0 {"id":"81a5b110-bd87-4bb1-b17b-ff110c4fe0b0","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84d814cc1 81a5b110-bd87-4bb1-b17b-ff110c4fe0b0 /tidb/cdc/default/default/upstream/7374684225589499271 {"id":7374684225589499271,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/81a5b110-bd87-4bb1-b17b-ff110c4fe0b0 {"id":"81a5b110-bd87-4bb1-b17b-ff110c4fe0b0","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84d814cc1 81a5b110-bd87-4bb1-b17b-ff110c4fe0b0 /tidb/cdc/default/default/upstream/7374684225589499271 {"id":7374684225589499271,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/81a5b110-bd87-4bb1-b17b-ff110c4fe0b0 {"id":"81a5b110-bd87-4bb1-b17b-ff110c4fe0b0","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84d814cc1 81a5b110-bd87-4bb1-b17b-ff110c4fe0b0 /tidb/cdc/default/default/upstream/7374684225589499271 {"id":7374684225589499271,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.cli.5913.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --server=127.0.0.1:8301 Create changefeed successfully! ID: 9457076d-a928-47b6-818a-63d75366eeb3 Info: {"upstream_id":7374684225589499271,"namespace":"default","id":"9457076d-a928-47b6-818a-63d75366eeb3","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T15:01:33.32615384+08:00","start_ts":450115008772702211,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115008772702211,"checkpoint_ts":450115008772702211,"checkpoint_time":"2024-05-30 15:01:33.182"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table changefeed_dup_error_restart.finish_mark_1 exists check diff successfully ***************** properties ***************** "requestdistribution"="uniform" "threadcount"="2" "insertproportion"="0" "mysql.host"="127.0.0.1" "dotransactions"="false" "readallfields"="true" "scanproportion"="0" "workload"="core" "recordcount"="10" "mysql.db"="changefeed_dup_error_restart" "updateproportion"="0" "mysql.user"="root" "mysql.port"="4000" "readproportion"="0" "operationcount"="0" ********************************************** Run finished, takes 5.101244ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2712.8, Avg(us): 914, Min(us): 519, Max(us): 2155, 95th(us): 3000, 99th(us): 3000 Confirm that you know what this command will do and use it at your own risk [Y/N] CDC service GC safepoint truncated in PD! PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... table processor_delay.t17 exists table processor_delay.t18 not exists for 1-th check, retry later [2024/05/30 15:01:33.737 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark12000] [2024/05/30 15:01:33.737 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark12000] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [2024/05/30 15:01:34.752 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark11000] [2024/05/30 15:01:34.929 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark11000] [ts=450115008967475205] [2024/05/30 15:01:34.929 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark11000] [2024/05/30 15:01:34.929 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=450115008967475205] [2024/05/30 15:01:34.933 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:01:34.938 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:01:35.031 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:01:35.036 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] table test.finish_mark not exists for 37-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' 1 run task successfully table processor_resolved_ts_fallback.t1 not exists for 1-th check, retry later Starting Upstream TiDB... + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11605.out cli unsafe reset --no-confirm --pd=http://127.0.0.1:2379 table processor_delay.t18 exists table processor_delay.t19 not exists for 1-th check, retry later Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '10.233.106.86' (111) reset and all metadata truncated in PD! PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... [2024/05/30 15:01:35.737 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark12000] table ddl_attributes.attributes_t1_new exists table ddl_attributes.finish_mark not exists for 1-th check, retry later table ddl_attributes.finish_mark not exists for 2-th check, retry later table processor_resolved_ts_fallback.t1 exists [Thu May 30 15:01:37 CST 2024] <<<<<< START cdc server in processor_resolved_ts_fallback case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.60076009.out server --log-file /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' table processor_delay.t19 exists table processor_delay.t20 not exists for 1-th check, retry later + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '10.233.106.86' (111) + set +x table test.finish_mark not exists for 38-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:01:37.829 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark12000] [2024/05/30 15:01:37.851 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark12000] [ts=450115009985118216] [2024/05/30 15:01:37.851 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark12000] [2024/05/30 15:01:37.851 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=450115009985118216] [2024/05/30 15:01:37.853 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:01:37.857 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:01:37.861 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:01:37.865 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:01:37.937 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark13000] [2024/05/30 15:01:37.937 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark13000] table ddl_attributes.finish_mark not exists for 3-th check, retry later table processor_delay.t20 exists table processor_delay.t21 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kafka_simple_basic/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:01:38 CST 2024] <<<<<< run test case kafka_simple_basic success! >>>>>> table test.finish_mark not exists for 39-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/changefeed_auto_stop/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '10.233.106.86' (111) [2024/05/30 15:01:39.937 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark13000] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > table ddl_attributes.finish_mark not exists for 4-th check, retry later + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11671.out cli unsafe resolve-lock --region=78 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... table consistent_replicate_storage_file.usertable exists table consistent_replicate_storage_file.check1 not exists for 1-th check, retry later table test.finish_mark not exists for 40-th check, retry later [2024/05/30 15:01:41.085 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc00184a1e0/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_storage_file.check1 not exists for 2-th check, retry later table processor_delay.t21 exists table processor_delay.t22 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2137aafc0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-hr7dq, pid:7087, start at 2024-05-30 15:01:40.712107919 +0800 CST m=+5.258844471 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:03:40.720 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:01:40.722 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:51:40.722 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '10.233.106.86' (111) [2024/05/30 15:01:41.951 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark13000] [2024/05/30 15:01:38.788 +08:00] [INFO] [main.go:99] ["running ddl test: 0 modifyColumnDefaultValueDDL1"] [2024/05/30 15:01:38.788 +08:00] [INFO] [main.go:99] ["running ddl test: 1 modifyColumnDefaultValueDDL2"] [2024/05/30 15:01:38.959 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs92513879_0bfa_45ea_b6b1_5dce716acfa4"] [2024/05/30 15:01:39.037 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs2428aead_0614_47a8_907e_03e664067e79"] [2024/05/30 15:01:39.072 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLsf44a5c86_b9fd_4a0a_8604_c99b246951cf"] [2024/05/30 15:01:39.118 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs46bc0e4b_e463_4392_952c_bb30ecd8f38c"] [2024/05/30 15:01:39.147 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs34e66ca4_c9d5_4857_a719_705fff04182c"] [2024/05/30 15:01:39.334 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs7ac982e2_9f7f_4733_b847_45594ed88435"] [2024/05/30 15:01:39.421 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLse92e028c_e0f1_43e0_9b3a_b6d437abf82c"] [2024/05/30 15:01:39.424 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLsb323f5a3_5a17_480f_baed_74a7894962a8"] [2024/05/30 15:01:39.538 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:01:39.637 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:01:39.835 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:01:39.918 +08:00] [INFO] [main.go:178] ["1 insert success: 100"] [2024/05/30 15:01:39.934 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:01:40.021 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:01:40.039 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:01:40.045 +08:00] [INFO] [main.go:178] ["1 insert success: 100"] [2024/05/30 15:01:40.129 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:01:40.131 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:01:40.146 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:01:40.319 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:01:40.331 +08:00] [INFO] [main.go:178] ["0 insert success: 100"] [2024/05/30 15:01:40.433 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:01:40.443 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:01:40.445 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:01:40.522 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:01:40.527 +08:00] [INFO] [main.go:178] ["0 insert success: 100"] [2024/05/30 15:01:40.528 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:01:40.621 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:01:40.723 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:01:40.736 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:01:40.950 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:01:41.026 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:01:41.122 +08:00] [INFO] [main.go:178] ["1 insert success: 200"] [2024/05/30 15:01:41.146 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:01:41.227 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:01:41.238 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:01:41.329 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:01:41.330 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:01:41.331 +08:00] [INFO] [main.go:178] ["1 insert success: 200"] [2024/05/30 15:01:41.426 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:01:41.444 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:01:41.520 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:01:41.527 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:01:41.726 +08:00] [INFO] [main.go:178] ["0 insert success: 200"] [2024/05/30 15:01:41.728 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:01:41.728 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:01:41.730 +08:00] [INFO] [main.go:199] ["0 delete success: 100"] [2024/05/30 15:01:41.732 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:01:41.821 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:01:41.920 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:01:41.923 +08:00] [INFO] [main.go:199] ["0 delete success: 100"] [2024/05/30 15:01:41.924 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:01:41.925 +08:00] [INFO] [main.go:178] ["0 insert success: 200"] [2024/05/30 15:01:42.141 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:01:42.142 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:01:42.222 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:01:42.235 +08:00] [INFO] [main.go:178] ["1 insert success: 300"] [2024/05/30 15:01:42.243 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:01:42.339 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:01:42.426 +08:00] [INFO] [main.go:178] ["1 insert success: 300"] [2024/05/30 15:01:42.431 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:01:42.446 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] table ddl_attributes.finish_mark not exists for 5-th check, retry later [2024/05/30 15:01:42.522 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:01:42.527 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:01:42.531 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11695.out cli unsafe resolve-lock --region=78 --ts=450115009874493443 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kafka_simple_basic_avro/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:01:42 CST 2024] <<<<<< run test case kafka_simple_basic_avro success! >>>>>> [2024/05/30 15:01:42.733 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:01:42.742 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:01:42.821 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:01:42.838 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:01:42.843 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:01:42.930 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:01:42.835 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0012d0000/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:01:42.838 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001a9c1e0/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] start tidb cluster in /tmp/tidb_cdc_test/changefeed_auto_stop Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... [2024/05/30 15:01:43.033 +08:00] [INFO] [main.go:178] ["0 insert success: 300"] [2024/05/30 15:01:43.120 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:01:43.138 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:01:43.139 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] table test.finish_mark not exists for 41-th check, retry later [2024/05/30 15:01:43.051 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001518000/127.0.0.1:2479] [attempt=0] [error="rpc error: code = NotFound desc = etcdserver: requested lease not found"] [2024/05/30 15:01:43.052 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0015423c0/127.0.0.1:2379] [attempt=0] [error="rpc error: code = NotFound desc = etcdserver: requested lease not found"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:01:43 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/9457076d-a928-47b6-818a-63d75366eeb3 {UpstreamID:7374684225589499271 Namespace:default ID:9457076d-a928-47b6-818a-63d75366eeb3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 15:01:33.32615384 +0800 CST StartTs:450115008772702211 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003a18870 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115008798916614} {CheckpointTs:450115009873707010 MinTableBarrierTs:450115009873707010 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/81a5b110-bd87-4bb1-b17b-ff110c4fe0b0 {"id":"81a5b110-bd87-4bb1-b17b-ff110c4fe0b0","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/b2bbf886-7139-4d7e-980e-bdc1e09f47d2 {"id":"b2bbf886-7139-4d7e-980e-bdc1e09f47d2","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84d814cc1 81a5b110-bd87-4bb1-b17b-ff110c4fe0b0 /tidb/cdc/default/__cdc_meta__/owner/22318fc84d814d33 b2bbf886-7139-4d7e-980e-bdc1e09f47d2 /tidb/cdc/default/default/changefeed/info/9457076d-a928-47b6-818a-63d75366eeb3 {"upstream-id":7374684225589499271,"namespace":"default","changefeed-id":"9457076d-a928-47b6-818a-63d75366eeb3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T15:01:33.32615384+08:00","start-ts":450115008772702211,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115008798916614} /tidb/cdc/default/default/changefeed/status/9457076d-a928-47b6-818a-63d75366eeb3 {"checkpoint-ts":450115009873707010,"min-table-barrier-ts":450115010083685336,"admin-job-type":0} /tidb/cdc/default/default/task/position/81a5b110-bd87-4bb1-b17b-ff110c4fe0b0/9457076d-a928-47b6-818a-63d75366eeb3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b2bbf886-7139-4d7e-980e-bdc1e09f47d2/9457076d-a928-47b6-818a-63d75366eeb3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684225589499271 {"id":7374684225589499271,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/9457076d-a928-47b6-818a-63d75366eeb3 {UpstreamID:7374684225589499271 Namespace:default ID:9457076d-a928-47b6-818a-63d75366eeb3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 15:01:33.32615384 +0800 CST StartTs:450115008772702211 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003a18870 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115008798916614} {CheckpointTs:450115009873707010 MinTableBarrierTs:450115009873707010 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/81a5b110-bd87-4bb1-b17b-ff110c4fe0b0 {"id":"81a5b110-bd87-4bb1-b17b-ff110c4fe0b0","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/b2bbf886-7139-4d7e-980e-bdc1e09f47d2 {"id":"b2bbf886-7139-4d7e-980e-bdc1e09f47d2","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84d814cc1 81a5b110-bd87-4bb1-b17b-ff110c4fe0b0 /tidb/cdc/default/__cdc_meta__/owner/22318fc84d814d33 b2bbf886-7139-4d7e-980e-bdc1e09f47d2 /tidb/cdc/default/default/changefeed/info/9457076d-a928-47b6-818a-63d75366eeb3 {"upstream-id":7374684225589499271,"namespace":"default","changefeed-id":"9457076d-a928-47b6-818a-63d75366eeb3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T15:01:33.32615384+08:00","start-ts":450115008772702211,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115008798916614} /tidb/cdc/default/default/changefeed/status/9457076d-a928-47b6-818a-63d75366eeb3 {"checkpoint-ts":450115009873707010,"min-table-barrier-ts":450115010083685336,"admin-job-type":0} /tidb/cdc/default/default/task/position/81a5b110-bd87-4bb1-b17b-ff110c4fe0b0/9457076d-a928-47b6-818a-63d75366eeb3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b2bbf886-7139-4d7e-980e-bdc1e09f47d2/9457076d-a928-47b6-818a-63d75366eeb3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684225589499271 + grep -q 'failed to get info:' {"id":7374684225589499271,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/9457076d-a928-47b6-818a-63d75366eeb3 {UpstreamID:7374684225589499271 Namespace:default ID:9457076d-a928-47b6-818a-63d75366eeb3 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 15:01:33.32615384 +0800 CST StartTs:450115008772702211 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003a18870 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115008798916614} {CheckpointTs:450115009873707010 MinTableBarrierTs:450115009873707010 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/81a5b110-bd87-4bb1-b17b-ff110c4fe0b0 {"id":"81a5b110-bd87-4bb1-b17b-ff110c4fe0b0","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/b2bbf886-7139-4d7e-980e-bdc1e09f47d2 {"id":"b2bbf886-7139-4d7e-980e-bdc1e09f47d2","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84d814cc1 81a5b110-bd87-4bb1-b17b-ff110c4fe0b0 /tidb/cdc/default/__cdc_meta__/owner/22318fc84d814d33 b2bbf886-7139-4d7e-980e-bdc1e09f47d2 /tidb/cdc/default/default/changefeed/info/9457076d-a928-47b6-818a-63d75366eeb3 {"upstream-id":7374684225589499271,"namespace":"default","changefeed-id":"9457076d-a928-47b6-818a-63d75366eeb3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T15:01:33.32615384+08:00","start-ts":450115008772702211,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115008798916614} /tidb/cdc/default/default/changefeed/status/9457076d-a928-47b6-818a-63d75366eeb3 {"checkpoint-ts":450115009873707010,"min-table-barrier-ts":450115010083685336,"admin-job-type":0} /tidb/cdc/default/default/task/position/81a5b110-bd87-4bb1-b17b-ff110c4fe0b0/9457076d-a928-47b6-818a-63d75366eeb3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b2bbf886-7139-4d7e-980e-bdc1e09f47d2/9457076d-a928-47b6-818a-63d75366eeb3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684225589499271 {"id":7374684225589499271,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [2024/05/30 15:01:43.223 +08:00] [INFO] [main.go:178] ["0 insert success: 300"] [2024/05/30 15:01:43.227 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:01:43.322 +08:00] [INFO] [main.go:178] ["1 insert success: 400"] [2024/05/30 15:01:43.421 +08:00] [INFO] [main.go:178] ["1 insert success: 400"] [2024/05/30 15:01:43.430 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:01:43.432 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:01:43.441 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:01:43.442 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] table processor_delay.t22 exists table processor_delay.t23 not exists for 1-th check, retry later [2024/05/30 15:01:43.519 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:01:43.546 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:01:43.730 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2137aafc0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-hr7dq, pid:7087, start at 2024-05-30 15:01:40.712107919 +0800 CST m=+5.258844471 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:03:40.720 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:01:40.722 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:51:40.722 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2137ae040014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-hr7dq, pid:7178, start at 2024-05-30 15:01:40.88293225 +0800 CST m=+5.356836224 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:03:40.890 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:01:40.865 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:51:40.865 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/http_proxies/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/http_proxies/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/http_proxies/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["10.233.106.86:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/http_proxies/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/http_proxies/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/common_1/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table processor_resolved_ts_fallback.t2 not exists for 1-th check, retry later [2024/05/30 15:01:43.832 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:01:43.832 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:01:43.844 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:01:44.025 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark13000] [2024/05/30 15:01:44.038 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark14000] [2024/05/30 15:01:44.038 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark14000] [2024/05/30 15:01:44.043 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark13000] [ts=450115011117056005] [2024/05/30 15:01:44.043 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark13000] [2024/05/30 15:01:44.044 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=450115011117056005] [2024/05/30 15:01:44.045 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:01:44.049 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:01:44.053 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:01:44.129 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:01:44.023 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:01:44.040 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:01:44.135 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:01:44.225 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:01:44.240 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:01:44.247 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:01:40.834 +08:00] [INFO] [case.go:115] ["sync updatePKUK take: 12.506152155s"] table consistent_replicate_storage_file.check1 not exists for 3-th check, retry later [2024/05/30 15:01:44.422 +08:00] [INFO] [main.go:178] ["1 insert success: 500"] [2024/05/30 15:01:44.424 +08:00] [INFO] [main.go:178] ["0 insert success: 400"] [2024/05/30 15:01:44.430 +08:00] [INFO] [main.go:199] ["0 delete success: 200"] [2024/05/30 15:01:44.440 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:01:44.444 +08:00] [INFO] [main.go:178] ["1 insert success: 500"] table ddl_attributes.finish_mark not exists for 6-th check, retry later + set +x % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 218 100 212 100 6 159k 4633 --:--:-- --:--:-- --:--:-- 207k { "error_msg": "[CDC:ErrAPIInvalidParam]invalid log level: json: cannot unmarshal string into Go value of type struct { Level string \"json:\\\"log_level\\\"\" }", "error_code": "CDC:ErrAPIInvalidParam" [2024/05/30 15:01:44.520 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:01:44.525 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:01:44.530 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:01:44.535 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:01:44.540 +08:00] [INFO] [main.go:199] ["0 delete success: 200"] [2024/05/30 15:01:44.541 +08:00] [INFO] [main.go:178] ["0 insert success: 400"] [2024/05/30 15:01:44.628 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:01:44.737 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:01:44.830 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:01:44.840 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:01:44.930 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:01:45.039 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:01:45.126 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:01:45.133 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:01:45.224 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kafka_simple_handle_key_only/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:01:45 CST 2024] <<<<<< run test case kafka_simple_handle_key_only success! >>>>>> [2024/05/30 15:01:45.319 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:01:45.329 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:01:45.437 +08:00] [INFO] [main.go:178] ["1 insert success: 600"] [2024/05/30 15:01:45.440 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:01:45.521 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:01:45.529 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:01:45.533 +08:00] [INFO] [main.go:178] ["1 insert success: 600"] table processor_delay.t23 exists table processor_delay.t24 not exists for 1-th check, retry later table test.finish_mark not exists for 42-th check, retry later [2024/05/30 15:01:45.630 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:01:45.635 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:01:45.732 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:01:45.818 +08:00] [INFO] [main.go:178] ["0 insert success: 500"] [2024/05/30 15:01:45.820 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:01:45.834 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] table processor_resolved_ts_fallback.t2 exists table processor_resolved_ts_fallback.t3 not exists for 1-th check, retry later [2024/05/30 15:01:46.039 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark14000] [2024/05/30 15:01:45.943 +08:00] [INFO] [main.go:178] ["0 insert success: 500"] [2024/05/30 15:01:45.943 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:01:46.026 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] table consistent_replicate_storage_file.check1 not exists for 4-th check, retry later + start_proxy + echo 'dumpling grpc packet to /tmp/tidb_cdc_test/http_proxies/test_proxy.log...' dumpling grpc packet to /tmp/tidb_cdc_test/http_proxies/test_proxy.log... + proxy_pid=8560 + echo 'proxy port: 12770' proxy port: 12770 + echo 'proxy pid: 8560' proxy pid: 8560 + sleep 5 + GO111MODULE=on + WORK_DIR=/tmp/tidb_cdc_test/http_proxies + go run /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/http_proxies/run-proxy.go --port=12770 go: downloading google.golang.org/grpc v1.64.0 go: downloading github.com/bradleyjkemp/grpc-tools v0.2.5 go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading go.uber.org/zap v1.27.0 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading github.com/improbable-eng/grpc-web v0.12.0 go: downloading golang.org/x/net v0.25.0 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading golang.org/x/sys v0.20.0 go: downloading github.com/gorilla/websocket v1.5.1 go: downloading github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f go: downloading github.com/rs/cors v1.7.0 go: downloading go.uber.org/multierr v1.11.0 go: downloading golang.org/x/text v0.15.0 go: downloading google.golang.org/protobuf v1.34.1 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240521202816-d264139d666e go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda [2024/05/30 15:01:46.121 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:01:46.132 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:01:46.146 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:01:46.234 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:01:46.244 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] start tidb cluster in /tmp/tidb_cdc_test/common_1 Starting Upstream PD... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [2024/05/30 15:01:46.425 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:01:46.434 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:01:46.435 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:01:46.524 +08:00] [INFO] [main.go:178] ["1 insert success: 700"] [2024/05/30 15:01:46.525 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:01:46.622 +08:00] [INFO] [main.go:178] ["1 insert success: 700"] Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... [2024/05/30 15:01:46.722 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:01:46.731 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:01:46.819 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:01:46.831 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:01:46.835 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:01:47.122 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:01:47.123 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:01:47.136 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:01:47.141 +08:00] [INFO] [main.go:178] ["0 insert success: 600"] [2024/05/30 15:01:47.146 +08:00] [INFO] [main.go:199] ["0 delete success: 300"] [2024/05/30 15:01:47.233 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:01:47.244 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:01:47.331 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] table processor_delay.t24 exists table processor_delay.t25 not exists for 1-th check, retry later [2024/05/30 15:01:47.343 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:01:47.429 +08:00] [INFO] [main.go:199] ["0 delete success: 300"] [2024/05/30 15:01:47.430 +08:00] [INFO] [main.go:178] ["0 insert success: 600"] [2024/05/30 15:01:47.524 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:01:47.524 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:01:47.546 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:01:47.619 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:01:47.639 +08:00] [INFO] [main.go:178] ["1 insert success: 800"] table test.finish_mark not exists for 43-th check, retry later table processor_resolved_ts_fallback.t3 exists cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^2$' VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2137f7100018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-cvg08, pid:6331, start at 2024-05-30 15:01:45.582162626 +0800 CST m=+10.484043050 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:03:45.589 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:01:45.589 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:51:45.589 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2137f7100018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-cvg08, pid:6331, start at 2024-05-30 15:01:45.582162626 +0800 CST m=+10.484043050 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:03:45.589 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:01:45.589 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:51:45.589 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2137fd48000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-cvg08, pid:6406, start at 2024-05-30 15:01:45.951834876 +0800 CST m=+10.795852421 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:03:45.959 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:01:45.938 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:51:45.938 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2024/05/30 15:01:47.738 +08:00] [INFO] [main.go:178] ["1 insert success: 800"] [2024/05/30 15:01:47.841 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] } % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 176 100 176 0 0 208k 0 --:--:-- --:--:-- --:--:-- 171k { "version": "v7.5.1-58-g8f92142d7", "git_hash": "8f92142d7874bfad2a8db6ca100604c383fca9df", "id": "796ba436-d650-4441-b5b8-7b2c2570114a", "pid": 10933, "is_owner": true }wait process cdc.test exit for 1-th time... 2 run task successfully table consistent_replicate_storage_file.check1 not exists for 5-th check, retry later [2024/05/30 15:01:47.922 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:01:47.923 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:01:47.925 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:01:48.026 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:01:48.143 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:01:48.143 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... [2024/05/30 15:01:48.240 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:01:48.329 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:01:48.338 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] wait process cdc.test exit for 2-th time... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:01:48.053 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark14000] [2024/05/30 15:01:48.142 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark14000] [ts=450115012165632005] [2024/05/30 15:01:48.142 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark14000] [2024/05/30 15:01:48.142 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=450115012165632005] [2024/05/30 15:01:48.145 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:01:48.148 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:01:48.153 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:01:48.225 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:01:48.432 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:01:48.437 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:01:48.629 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:01:48.635 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:01:48.640 +08:00] [INFO] [main.go:178] ["0 insert success: 700"] [2024/05/30 15:01:48.642 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:01:48.719 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:01:48.720 +08:00] [INFO] [main.go:178] ["1 insert success: 900"] [2024/05/30 15:01:48.833 +08:00] [INFO] [main.go:178] ["1 insert success: 900"] [2024/05/30 15:01:48.927 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:01:48.932 +08:00] [INFO] [main.go:178] ["0 insert success: 700"] [2024/05/30 15:01:48.938 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] wait process cdc.test exit for 3-th time... [2024/05/30 15:01:49.020 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:01:49.025 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:01:49.040 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:01:49.040 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark15000] [2024/05/30 15:01:49.040 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark15000] table processor_delay.t25 exists [2024/05/30 15:01:49.225 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:01:49.228 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:01:49.334 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:01:49.336 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:01:49.438 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kafka_simple_handle_key_only_avro/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:01:48 CST 2024] <<<<<< run test case kafka_simple_handle_key_only_avro success! >>>>>> table test.finish_mark not exists for 44-th check, retry later table processor_delay.t26 not exists for 1-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:01:49 CST 2024] <<<<<< run test case cli success! >>>>>> [Thu May 30 15:01:49 CST 2024] <<<<<< START cdc server in changefeed_resume_with_checkpoint_ts case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_resume_with_checkpoint_ts.77907792.out server --log-file /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/05/30 15:01:49.523 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:01:49.620 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] table ddl_attributes.finish_mark exists check diff successfully [2024/05/30 15:01:49.730 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:01:49.743 +08:00] [INFO] [main.go:178] ["1 insert success: 1000"] [2024/05/30 15:01:49.743 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:01:49.743 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:01:49.829 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:01:49.941 +08:00] [INFO] [main.go:178] ["1 insert success: 1000"] table consistent_replicate_storage_file.check1 not exists for 6-th check, retry later [2024/05/30 15:01:50.035 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:01:50.123 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:01:50.125 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:01:50.135 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:01:50.142 +08:00] [INFO] [main.go:178] ["0 insert success: 800"] [2024/05/30 15:01:50.147 +08:00] [INFO] [main.go:199] ["0 delete success: 400"] wait process cdc.test exit for 1-th time... [2024/05/30 15:01:50.230 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:01:50.330 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:01:50.334 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:01:50.422 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:01:50.436 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:01:50.443 +08:00] [INFO] [main.go:199] ["0 delete success: 400"] [2024/05/30 15:01:50.444 +08:00] [INFO] [main.go:178] ["0 insert success: 800"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:01:50.538 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:01:50.622 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:01:50.639 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] Verifying downstream PD is started... wait process cdc.test exit for 2-th time... [2024/05/30 15:01:50.746 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:01:50.832 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:01:50.842 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:01:50.928 +08:00] [INFO] [main.go:178] ["1 insert success: 1100"] [2024/05/30 15:01:50.929 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:01:51.041 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark15000] killing cdc1: 5863 cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' [2024/05/30 15:01:51.136 +08:00] [INFO] [main.go:178] ["1 insert success: 1100"] [2024/05/30 15:01:51.233 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:01:51.234 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:01:51.235 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 15:01:51 CST 2024] <<<<<< run test case ddl_attributes success! >>>>>> table test.finish_mark not exists for 45-th check, retry later table processor_delay.t26 exists table processor_delay.t27 not exists for 1-th check, retry later [2024/05/30 15:01:51.332 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] + export http_proxy=http://127.0.0.1:12770 + http_proxy=http://127.0.0.1:12770 + export https_proxy=http://127.0.0.1:12770 + https_proxy=http://127.0.0.1:12770 + echo 'try to connect pd cluster via proxy, pd addr: 10.233.106.86:2379' try to connect pd cluster via proxy, pd addr: 10.233.106.86:2379 + ensure 10 curl http://10.233.106.86:2379/ curl http://10.233.106.86:2379/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to 127.0.0.1:12770; Connection refused run task failed 1-th time, retry later [2024/05/30 15:01:51.532 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:01:51.630 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:01:51.636 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:01:51.746 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:01:51.834 +08:00] [INFO] [main.go:178] ["0 insert success: 900"] [2024/05/30 15:01:51.838 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:01:51.842 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:01:51.935 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:01:52.026 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:01:52.030 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:01:52.119 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:01:52.126 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:01:52.139 +08:00] [INFO] [main.go:178] ["0 insert success: 900"] [2024/05/30 15:01:52.236 +08:00] [INFO] [main.go:178] ["1 insert success: 1200"] [2024/05/30 15:01:52.325 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:01:52.435 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/05/30 15:01:52.439 +08:00] [INFO] [main.go:178] ["1 insert success: 1200"] [2024/05/30 15:01:52.444 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/05/30 15:01:52.533 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] table consistent_replicate_storage_file.check1 not exists for 7-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kafka_simple_claim_check/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:01:52 CST 2024] <<<<<< run test case kafka_simple_claim_check success! >>>>>> [2024/05/30 15:01:52.628 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:01:52.723 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/05/30 15:01:52.745 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] curl http://10.233.106.86:2379/ [2024/05/30 15:01:52.838 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:01:53.018 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/05/30 15:01:53.020 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to 127.0.0.1:12770; Connection refused run task failed 2-th time, retry later [2024/05/30 15:01:53.126 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/05/30 15:01:53.132 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:01:53.143 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] table test.finish_mark not exists for 46-th check, retry later [2024/05/30 15:01:53.319 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/05/30 15:01:53.319 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/05/30 15:01:53.330 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/05/30 15:01:53.434 +08:00] [INFO] [main.go:178] ["0 insert success: 1000"] [2024/05/30 15:01:53.439 +08:00] [INFO] [main.go:199] ["0 delete success: 500"] [2024/05/30 15:01:53.521 +08:00] [INFO] [main.go:178] ["1 insert success: 1300"] [2024/05/30 15:01:53.054 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark15000] [2024/05/30 15:01:53.070 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark15000] [ts=450115013628919810] [2024/05/30 15:01:53.070 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark15000] [2024/05/30 15:01:53.070 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=450115013628919810] [2024/05/30 15:01:53.289 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:01:53.293 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:01:53.297 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:01:53.327 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:01:53.625 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/05/30 15:01:53.731 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/05/30 15:01:53.732 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/05/30 15:01:53.735 +08:00] [INFO] [main.go:178] ["1 insert success: 1300"] table processor_delay.t27 exists table processor_delay.t28 not exists for 1-th check, retry later [2024/05/30 15:01:53.826 +08:00] [INFO] [main.go:199] ["0 delete success: 500"] [2024/05/30 15:01:53.827 +08:00] [INFO] [main.go:178] ["0 insert success: 1000"] [2024/05/30 15:01:53.842 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/05/30 15:01:54.034 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/05/30 15:01:54.125 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/05/30 15:01:54.135 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/05/30 15:01:54.138 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/05/30 15:01:54.230 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/05/30 15:01:54.319 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/05/30 15:01:54.328 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/05/30 15:01:54.340 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/05/30 15:01:54.424 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/05/30 15:01:54.426 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/05/30 15:01:54.544 +08:00] [INFO] [main.go:178] ["1 insert success: 1400"] [2024/05/30 15:01:54.622 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/05/30 15:01:54.734 +08:00] [INFO] [main.go:178] ["1 insert success: 1400"] [2024/05/30 15:01:54.736 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] [2024/05/30 15:01:54.834 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:01:54.934 +08:00] [INFO] [main.go:178] ["0 insert success: 1100"] [2024/05/30 15:01:54.945 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/05/30 15:01:55.039 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kafka_simple_claim_check_avro/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:01:55 CST 2024] <<<<<< run test case kafka_simple_claim_check_avro success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 1 -eq 50 ']' + sleep 3 [2024/05/30 15:01:55.053 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc00184a1e0/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:01:55.053 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001542d20/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:01:55.127 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/05/30 15:01:55.141 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] [2024/05/30 15:01:55.148 +08:00] [INFO] [main.go:178] ["0 insert success: 1100"] [2024/05/30 15:01:55.231 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] [2024/05/30 15:01:55.248 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] [2024/05/30 15:01:55.249 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/05/30 15:01:55.321 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] [2024/05/30 15:01:55.340 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] [2024/05/30 15:01:55.054 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc000e0eb40/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:01:55.344 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0019501e0/127.0.0.1:2479] [attempt=0] [error="rpc error: code = NotFound desc = etcdserver: requested lease not found"] [2024/05/30 15:01:55.346 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0015423c0/127.0.0.1:2379] [attempt=0] [error="rpc error: code = NotFound desc = etcdserver: requested lease not found"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 2 -eq 50 ']' + sleep 3 [2024/05/30 15:01:55.519 +08:00] [INFO] [main.go:178] ["1 insert success: 1500"] Error: Get "http://127.0.0.1:8301/api/v2/captures": read tcp 127.0.0.1:59720->127.0.0.1:8301: read: connection reset by peer run task failed 1-th time, retry later table test.finish_mark not exists for 47-th check, retry later table processor_delay.t28 exists table processor_delay.t29 not exists for 1-th check, retry later [2024/05/30 15:01:55.728 +08:00] [INFO] [main.go:178] ["1 insert success: 1500"] [2024/05/30 15:01:55.346 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001544000/127.0.0.1:2379] [attempt=0] [error="rpc error: code = NotFound desc = etcdserver: requested lease not found"] Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [2024/05/30 15:01:55.935 +08:00] [INFO] [main.go:178] ["73 insert success: 1500"] [2024/05/30 15:01:55.947 +08:00] [INFO] [main.go:178] ["73 insert success: 1500"] [2024/05/30 15:01:56.033 +08:00] [INFO] [main.go:178] ["73 insert success: 1500"] [2024/05/30 15:01:56.144 +08:00] [INFO] [main.go:178] ["73 insert success: 1500"] [2024/05/30 15:01:56.152 +08:00] [INFO] [main.go:178] ["72 insert success: 1600"] [2024/05/30 15:01:56.154 +08:00] [INFO] [main.go:178] ["0 insert success: 1200"] [2024/05/30 15:01:56.155 +08:00] [INFO] [main.go:178] ["72 insert success: 1600"] [2024/05/30 15:01:56.222 +08:00] [INFO] [main.go:199] ["0 delete success: 600"] [2024/05/30 15:01:56.250 +08:00] [INFO] [main.go:178] ["72 insert success: 1600"] [2024/05/30 15:01:56.318 +08:00] [INFO] [main.go:178] ["1 insert success: 1600"] [2024/05/30 15:01:56.344 +08:00] [INFO] [main.go:199] ["0 delete success: 600"] [2024/05/30 15:01:56.345 +08:00] [INFO] [main.go:178] ["0 insert success: 1200"] [2024/05/30 15:01:56.455 +08:00] [INFO] [main.go:178] ["1 insert success: 1600"] [2024/05/30 15:01:56.621 +08:00] [INFO] [main.go:178] ["73 insert success: 1600"] table consistent_replicate_storage_file.check1 exists table consistent_replicate_storage_file.t2 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:01:56.721 +08:00] [INFO] [main.go:178] ["73 insert success: 1600"] [2024/05/30 15:01:56.736 +08:00] [INFO] [main.go:178] ["72 insert success: 1700"] [2024/05/30 15:01:56.766 +08:00] [INFO] [main.go:178] ["72 insert success: 1700"] [2024/05/30 15:01:56.950 +08:00] [INFO] [main.go:178] ["1 insert success: 1700"] [2024/05/30 15:01:56.973 +08:00] [INFO] [main.go:178] ["73 insert success: 1700"] [2024/05/30 15:01:56.980 +08:00] [INFO] [main.go:178] ["0 insert success: 1300"] [2024/05/30 15:01:57.128 +08:00] [INFO] [main.go:178] ["72 insert success: 1800"] [2024/05/30 15:01:57.131 +08:00] [INFO] [main.go:178] ["73 insert success: 1700"] [2024/05/30 15:01:57.163 +08:00] [INFO] [main.go:178] ["72 insert success: 1800"] [2024/05/30 15:01:57.322 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLsa14e6960_f5da_4a03_9c1c_3223ba3b2949"] [2024/05/30 15:01:57.323 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLsbe5530ef_2ffb_4a31_8226_656fd3a32f2f"] [2024/05/30 15:01:57.330 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs0b2d38f9_112f_4f1f_a23e_14bc80eaca60"] [2024/05/30 15:01:57.341 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs8cffa0e6_ad2e_4879_ae92_c771842bacf9"] [2024/05/30 15:01:57.350 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs77195e8f_b6d6_4f91_8e95_c019c48b49a3"] cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused run task failed 2-th time, retry later [2024/05/30 15:01:57.430 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLsc91d14d0_592d_46d8_8a24_e25eaba9a460"] [2024/05/30 15:01:57.438 +08:00] [INFO] [main.go:178] ["1 insert success: 1800"] table test.finish_mark not exists for 48-th check, retry later table processor_delay.t29 exists table processor_delay.t30 not exists for 1-th check, retry later curl http://10.233.106.86:2379/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to 127.0.0.1:12770; Connection refused run task failed 3-th time, retry later [2024/05/30 15:01:55.254 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001a9c1e0/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:01:57.743 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs76b98b41_2e14_4bc2_b338_d2e001aaeced"] Starting Upstream TiDB... [2024/05/30 15:01:57.822 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs3bd74174_50c2_4dae_bd0e_fbfe0a3eedba"] Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community [2024/05/30 15:01:57.826 +08:00] [INFO] [main.go:199] ["0 delete success: 700"] Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 [2024/05/30 15:01:57.827 +08:00] [INFO] [main.go:178] ["0 insert success: 1400"] Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_storage_file.t2 not exists for 2-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/canal_json_adapter_compatibility/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:01:58 CST 2024] <<<<<< run test case canal_json_adapter_compatibility success! >>>>>> [2024/05/30 15:01:58.231 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:01:58.237 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:01:58.241 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:01:58.318 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:01:58.321 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:01:58.322 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:01:58.323 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:01:58.329 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:01:58.332 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:01:58.334 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:01:58.336 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:01:58.349 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:01:58.350 +08:00] [INFO] [main.go:178] ["1 insert success: 1900"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:01:58.727 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:01:58.737 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:01:58.740 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:01:58 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3357343a-272a-492a-8579-fa2918157468 {"id":"3357343a-272a-492a-8579-fa2918157468","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84dbce837 3357343a-272a-492a-8579-fa2918157468 /tidb/cdc/default/default/upstream/7374684287058025977 {"id":7374684287058025977,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3357343a-272a-492a-8579-fa2918157468 {"id":"3357343a-272a-492a-8579-fa2918157468","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84dbce837 3357343a-272a-492a-8579-fa2918157468 /tidb/cdc/default/default/upstream/7374684287058025977 {"id":7374684287058025977,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3357343a-272a-492a-8579-fa2918157468 {"id":"3357343a-272a-492a-8579-fa2918157468","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84dbce837 3357343a-272a-492a-8579-fa2918157468 /tidb/cdc/default/default/upstream/7374684287058025977 {"id":7374684287058025977,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [2024/05/30 15:01:58.820 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:01:59.141 +08:00] [INFO] [main.go:178] ["0 insert success: 1500"] [2024/05/30 15:01:59.224 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:01:59.231 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:01:59.238 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:01:59.240 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:01:59.244 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:01:59.247 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] table test.table1 not exists for 1-th check, retry later [2024/05/30 15:01:59.341 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:01:59.342 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:01:59.346 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:01:59.420 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:01:59.421 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:01:59.450 +08:00] [INFO] [main.go:178] ["1 insert success: 2000"] [2024/05/30 15:01:59.451 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] table test.finish_mark not exists for 49-th check, retry later table processor_delay.t30 exists table processor_delay.t31 not exists for 1-th check, retry later [2024/05/30 15:01:59.633 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:01:59.723 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) time="2024-05-30T15:01:59+08:00" level=info msg="Listening on 127.0.0.1:12770" time="2024-05-30T15:01:59+08:00" level=info msg="Not intercepting TLS connections" ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_storage_file.t2 not exists for 3-th check, retry later [2024/05/30 15:02:00.526 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:02:00.730 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:02:01.219 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc already exists) table test.finish_mark not exists for 50-th check, retry later [2024/05/30 15:02:01.320 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:02:01.435 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [Pipeline] // cache table processor_delay.t31 exists table processor_delay.t32 not exists for 1-th check, retry later [Pipeline] } cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' 1 run task successfully [Pipeline] // dir ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] } [2024/05/30 15:02:01.529 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:02:01.621 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:02:01.631 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/canal_json_basic/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:02:01 CST 2024] <<<<<< run test case canal_json_basic success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/savepoint/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] // withCredentials table test.table1 exists table test.table2 not exists for 1-th check, retry later check diff failed 1-th time, retry later [Pipeline] } [2024/05/30 15:02:01.928 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [2024/05/30 15:02:02.020 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:02:02.023 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:02:02.023 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:02:02.024 +08:00] [INFO] [main.go:199] ["0 delete success: 800"] [2024/05/30 15:02:02.024 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:02:02.025 +08:00] [INFO] [main.go:178] ["0 insert success: 1600"] [2024/05/30 15:02:02.127 +08:00] [INFO] [main.go:178] ["1 insert success: 2100"] [2024/05/30 15:02:02.128 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [2024/05/30 15:02:02.325 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:02:02.328 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] table consistent_replicate_storage_file.t2 not exists for 4-th check, retry later [Pipeline] // node [Pipeline] } [2024/05/30 15:02:02.636 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:02:02.730 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [2024/05/30 15:02:02.933 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:02.936 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:03.021 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:03.033 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:03.035 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [Pipeline] // stage ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] } [2024/05/30 15:02:03.042 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:03.222 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:03.227 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:03.233 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:03.236 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:03.236 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] table test.finish_mark not exists for 51-th check, retry later [2024/05/30 15:02:03.324 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:03.337 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:03.337 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:03.339 +08:00] [INFO] [main.go:178] ["1 insert success: 2200"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) curl http://10.233.106.86:2379/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 19 100 19 0 0 17086 0 --:--:-- --:--:-- --:--:-- 19000 404 page not found run task successfully + echo started proxy pid: 8560 started proxy pid: 8560 + echo started proxy at port: 12770 started proxy at port: 12770 + cd /tmp/tidb_cdc_test/http_proxies ++ run_cdc_cli_tso_query 10.233.106.86 2379 + pd_host=10.233.106.86 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://10.233.106.86:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_proxies.cli.12020.out cli tso query --pd=http://10.233.106.86:2379 table test.table2 exists check diff failed 1-th time, retry later [2024/05/30 15:02:03.629 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:03.726 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] table processor_delay.t32 exists table processor_delay.t33 not exists for 1-th check, retry later [2024/05/30 15:02:03.934 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:04.025 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:04.119 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:04.132 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:04.136 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:04.139 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] table consistent_replicate_storage_file.t2 not exists for 5-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... [2024/05/30 15:02:04.330 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:04.335 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:04.338 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:04.340 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:04.341 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:04.433 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:04.435 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:04.439 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:04.636 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:04.733 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:04.830 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x failed proxying s2c rpc error: code = Canceled desc = context canceled failed proxying s2c rpc error: code = Canceled desc = context canceled + tso='450115016647507972 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115016647507972 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=450115016647507972 + echo 'query start ts: 450115016647507972' query start ts: 450115016647507972 + run_cdc_server --workdir /tmp/tidb_cdc_test/http_proxies --binary cdc.test [Thu May 30 15:02:04 CST 2024] <<<<<< START cdc server in http_proxies case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_proxies.1206012062.out server --log-file /tmp/tidb_cdc_test/http_proxies/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_proxies/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to proxy 127.0.0.1 port 12770 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 12770 (#0) > GET http://127.0.0.1:8300/debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > Proxy-Connection: Keep-Alive > < HTTP/1.1 502 Bad Gateway < Date: Thu, 30 May 2024 07:02:04 GMT < Content-Length: 0 < * Connection #0 to host 127.0.0.1 left intact + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 2-th time... [2024/05/30 15:02:04.839 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:02:05.018 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/multi_topics/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:02:04 CST 2024] <<<<<< run test case multi_topics success! >>>>>> failed proxying s2c rpc error: code = Canceled desc = context canceled [2024/05/30 15:02:05.127 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:02:05.128 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:02:05.129 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:02:05.441 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:02:05.526 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:02:05.530 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:02:05.534 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:02:05.534 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] wait process cdc.test exit for 3-th time... table changefeed_dup_error_restart.finish_mark_2 exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:02:04 CST 2024] <<<<<< run test case changefeed_dup_error_restart success! >>>>>> [2024/05/30 15:02:05.622 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:02:05.622 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:02:05.628 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:02:05.825 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] table test.finish_mark not exists for 52-th check, retry later table processor_delay.t33 exists table processor_delay.t34 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21390a600013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-9tzt0, pid:6408, start at 2024-05-30 15:02:03.192335065 +0800 CST m=+5.751089286 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:03.202 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:03.160 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:03.160 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21390a600013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-9tzt0, pid:6408, start at 2024-05-30 15:02:03.192335065 +0800 CST m=+5.751089286 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:03.202 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:03.160 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:03.160 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21390c940013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-9tzt0, pid:6482, start at 2024-05-30 15:02:03.332120927 +0800 CST m=+5.797954217 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:03.339 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:03.301 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:03.301 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/common_1/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/common_1/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/common_1/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/common_1/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/common_1/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2024/05/30 15:02:05.921 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:02:05.939 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:05.945 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:06.035 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:06.045 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:06.048 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:06.051 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:02:05 CST 2024] <<<<<< run test case processor_resolved_ts_fallback success! >>>>>> table consistent_replicate_storage_file.t2 exists check diff successfully [2024/05/30 15:02:06.520 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:02:06.525 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:02:06.527 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:06.530 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:06.531 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:02:06.535 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:02:06.622 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] Resume changefeed with checkpointTs3 450115017464872965 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:02:06.722 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:02:06.921 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] start tidb cluster in /tmp/tidb_cdc_test/savepoint Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... table test.table5 not exists for 1-th check, retry later [2024/05/30 15:02:07.022 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:02:07.032 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:02:07.040 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:02:07.134 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] check diff successfully [2024/05/30 15:02:07.222 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:02:07.222 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:02:07.227 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:02:07.425 +08:00] [INFO] [main.go:88] ["testGetDefaultValue take 28.63682732s"] wait process cdc.test exit for 1-th time... [2024/05/30 15:02:07.533 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:02:07.534 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:02:07.534 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:02:07.535 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:02:07.625 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:02:07.631 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] table test.finish_mark not exists for 53-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to proxy 127.0.0.1 port 12770 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 12770 (#0) > GET http://127.0.0.1:8300/debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > Proxy-Connection: Keep-Alive > < HTTP/1.1 200 OK < Content-Length: 617 < Content-Type: text/plain; charset=utf-8 < Date: Thu, 30 May 2024 07:02:07 GMT < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/06604b76-c4d7-4627-acff-397b19b12da8 {"id":"06604b76-c4d7-4627-acff-397b19b12da8","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1e9f8fc84dc08a01 06604b76-c4d7-4627-acff-397b19b12da8 /tidb/cdc/default/default/upstream/7374684302583465386 {"id":7374684302583465386,"pd-endpoints":"http://127.0.0.1:2379,http://10.233.106.86:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/06604b76-c4d7-4627-acff-397b19b12da8 {"id":"06604b76-c4d7-4627-acff-397b19b12da8","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1e9f8fc84dc08a01 06604b76-c4d7-4627-acff-397b19b12da8 /tidb/cdc/default/default/upstream/7374684302583465386 {"id":7374684302583465386,"pd-endpoints":"http://127.0.0.1:2379,http://10.233.106.86:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/06604b76-c4d7-4627-acff-397b19b12da8 {"id":"06604b76-c4d7-4627-acff-397b19b12da8","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1e9f8fc84dc08a01 06604b76-c4d7-4627-acff-397b19b12da8 /tidb/cdc/default/default/upstream/7374684302583465386 {"id":7374684302583465386,"pd-endpoints":"http://127.0.0.1:2379,http://10.233.106.86:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + echo started cdc server successfully started cdc server successfully + SINK_URI=blackhole:/// + run_cdc_cli changefeed create --start-ts=450115016647507972 --sink-uri=blackhole:/// + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_proxies.cli.12114.out cli changefeed create --start-ts=450115016647507972 --sink-uri=blackhole:/// table processor_delay.t34 exists table processor_delay.t35 not exists for 1-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.common_1.cli.7792.out cli tso query --pd=http://127.0.0.1:2379 [2024/05/30 15:02:07.722 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:02:07.821 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] wait process cdc.test exit for 2-th time... [2024/05/30 15:02:08.132 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] failed proxying s2c rpc error: code = Canceled desc = context canceled Create changefeed successfully! ID: 15394dc7-3157-4849-8e3c-51ebeb428d0e Info: {"upstream_id":7374684302583465386,"namespace":"default","id":"15394dc7-3157-4849-8e3c-51ebeb428d0e","sink_uri":"blackhole:///","create_time":"2024-05-30T15:02:08.150944314+08:00","start_ts":450115016647507972,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115016647507972,"checkpoint_ts":450115016647507972,"checkpoint_time":"2024-05-30 15:02:03.222"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... Verifying downstream PD is started... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/avro_basic/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:02:08 CST 2024] <<<<<< run test case avro_basic success! >>>>>> [2024/05/30 15:02:06.636 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark16000] [2024/05/30 15:02:06.637 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark16000] [2024/05/30 15:02:08.237 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:02:08.239 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:08.329 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:08.348 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:08.425 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:08.432 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:08.435 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] wait process cdc.test exit for 3-th time... [2024/05/30 15:02:08.637 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark16000] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:02:08.633 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:08.634 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:08.646 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:02:08.718 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:02:08.734 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:02:08.736 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:02:08.740 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:02:08.732 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark16000] [ts=450115017932800005] [2024/05/30 15:02:08.732 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark16000] [2024/05/30 15:02:08.732 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=450115017932800005] [2024/05/30 15:02:08.738 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:02:08.742 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:02:08.749 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:02:08.839 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:02:08.831 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:02:08 CST 2024] <<<<<< START cdc server in consistent_replicate_storage_file case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=return(true)' + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_storage_file.1135511357.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/05/30 15:02:09.238 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [2024/05/30 15:02:09.339 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:02:09.341 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:02:09.429 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] table test.table5 not exists for 2-th check, retry later table test.finish_mark not exists for 54-th check, retry later table processor_delay.t35 exists table processor_delay.t36 not exists for 1-th check, retry later + set +x + sleep 5 [2024/05/30 15:02:09.519 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:02:09.531 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:02:09.539 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:02:09.539 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:02:09.733 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:02:09.735 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] + set +x + tso='450115017955082241 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115017955082241 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 15:02:09 CST 2024] <<<<<< START cdc server in common_1 case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.common_1.78257827.out server --log-file /tmp/tidb_cdc_test/common_1/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/common_1/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/05/30 15:02:09.820 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:09.827 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:09.929 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:09.935 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:09.936 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:10.034 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213964000006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6w171, pid:9491, start at 2024-05-30 15:02:08.9030999 +0800 CST m=+20.554943869 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:08.914 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:08.896 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:08.896 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213964000006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6w171, pid:9491, start at 2024-05-30 15:02:08.9030999 +0800 CST m=+20.554943869 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:08.914 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:08.896 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:08.896 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213968b00011 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6w171, pid:9579, start at 2024-05-30 15:02:09.217650311 +0800 CST m=+20.768694692 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:09.225 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:09.196 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:09.196 +0800 All versions after safe point can be accessed. (DO NOT EDIT) [2024/05/30 15:02:10.528 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:10.624 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:08.939 +08:00] [WARN] [diff.go:179] ["table struct is not equal"] [reason="column num not equal, one is 5 another is 4"] Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2024/05/30 15:02:10.628 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:10.725 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:10.824 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:10.833 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:10.840 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:10.842 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:11.035 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:11.035 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:02:11.118 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:02:11.122 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:11.231 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:02:11.321 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:02:11.325 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:02:11.336 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:02:11.032 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark17000] [2024/05/30 15:02:11.032 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark17000] table test.table5 exists table test.table6 not exists for 1-th check, retry later table test.finish_mark not exists for 55-th check, retry later table processor_delay.t36 exists =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/canal_json_handle_key_only/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:02:11 CST 2024] <<<<<< run test case canal_json_handle_key_only success! >>>>>> Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t37 not exists for 1-th check, retry later [2024/05/30 15:02:11.821 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:02:11.835 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:02:11.839 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:02:11.933 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:02:12.025 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:02:12.026 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:02:12.037 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:02:12.037 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:02:12.161 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:02:12.166 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:02:12.171 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:02:12.224 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:02:12.319 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:02:12.329 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:02:12.421 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:02:13.025 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:02:12 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/45de9274-aad7-4260-9b54-acd46ad16406 {"id":"45de9274-aad7-4260-9b54-acd46ad16406","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84e0153db 45de9274-aad7-4260-9b54-acd46ad16406 /tidb/cdc/default/default/upstream/7374684373662834471 {"id":7374684373662834471,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/45de9274-aad7-4260-9b54-acd46ad16406 {"id":"45de9274-aad7-4260-9b54-acd46ad16406","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84e0153db 45de9274-aad7-4260-9b54-acd46ad16406 /tidb/cdc/default/default/upstream/7374684373662834471 {"id":7374684373662834471,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/45de9274-aad7-4260-9b54-acd46ad16406 {"id":"45de9274-aad7-4260-9b54-acd46ad16406","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84e0153db 45de9274-aad7-4260-9b54-acd46ad16406 /tidb/cdc/default/default/upstream/7374684373662834471 {"id":7374684373662834471,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.common_1.cli.7872.out cli changefeed create --start-ts=450115017955082241 --sink-uri=mysql://root@127.0.0.1:3306/ [2024/05/30 15:02:13.047 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:02:13.135 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/05/30 15:02:13.135 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/05/30 15:02:13.142 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/05/30 15:02:13.143 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/05/30 15:02:13.232 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/05/30 15:02:13.233 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/05/30 15:02:13.234 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/05/30 15:02:13.256 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > [2024/05/30 15:02:13.032 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark17000] Create changefeed successfully! ID: 41c1abd7-3317-4a8e-b472-e82e80d73733 Info: {"upstream_id":7374684373662834471,"namespace":"default","id":"41c1abd7-3317-4a8e-b472-e82e80d73733","sink_uri":"mysql://root@127.0.0.1:3306/","create_time":"2024-05-30T15:02:13.21667162+08:00","start_ts":450115017955082241,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115017955082241,"checkpoint_ts":450115017955082241,"checkpoint_time":"2024-05-30 15:02:08.210"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table test.table6 exists check diff failed 1-th time, retry later ***************** properties ***************** "workload"="core" "mysql.port"="4000" "readproportion"="0" "dotransactions"="false" "mysql.host"="127.0.0.1" "updateproportion"="0" "recordcount"="20" "mysql.db"="changefeed_auto_stop_1" "scanproportion"="0" "insertproportion"="0" "threadcount"="4" "mysql.user"="root" "readallfields"="true" "operationcount"="0" "requestdistribution"="uniform" ********************************************** Run finished, takes 13.835969ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2383.5, Avg(us): 2620, Min(us): 1479, Max(us): 5361, 95th(us): 6000, 99th(us): 6000 table test.finish_mark not exists for 56-th check, retry later [2024/05/30 15:02:13.321 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/05/30 15:02:13.328 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] ***************** properties ***************** "mysql.user"="root" "requestdistribution"="uniform" "threadcount"="4" "recordcount"="20" "operationcount"="0" "readallfields"="true" "readproportion"="0" "dotransactions"="false" "mysql.host"="127.0.0.1" "workload"="core" "scanproportion"="0" "updateproportion"="0" "mysql.port"="4000" "mysql.db"="changefeed_auto_stop_2" "insertproportion"="0" ********************************************** Run finished, takes 13.875352ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2266.5, Avg(us): 2435, Min(us): 1420, Max(us): 5001, 95th(us): 5000, 99th(us): 6000 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t37 exists table processor_delay.t38 not exists for 1-th check, retry later [2024/05/30 15:02:13.735 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] ***************** properties ***************** "mysql.port"="4000" "requestdistribution"="uniform" "operationcount"="0" "mysql.host"="127.0.0.1" "recordcount"="20" "scanproportion"="0" "updateproportion"="0" "threadcount"="4" "dotransactions"="false" "readproportion"="0" "readallfields"="true" "workload"="core" "mysql.db"="changefeed_auto_stop_3" "insertproportion"="0" "mysql.user"="root" ********************************************** Run finished, takes 12.003623ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2916.4, Avg(us): 2240, Min(us): 1187, Max(us): 5325, 95th(us): 6000, 99th(us): 6000 [2024/05/30 15:02:13.934 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] ***************** properties ***************** "scanproportion"="0" "dotransactions"="false" "recordcount"="20" "mysql.port"="4000" "readallfields"="true" "mysql.db"="changefeed_auto_stop_4" "updateproportion"="0" "requestdistribution"="uniform" "workload"="core" "mysql.user"="root" "readproportion"="0" "threadcount"="4" "insertproportion"="0" "mysql.host"="127.0.0.1" "operationcount"="0" ********************************************** Run finished, takes 17.01629ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2197.7, Avg(us): 3117, Min(us): 1271, Max(us): 7774, 95th(us): 8000, 99th(us): 8000 [Thu May 30 15:02:13 CST 2024] <<<<<< START cdc server in changefeed_auto_stop case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_auto_stop.1094510947.out server --log-file /tmp/tidb_cdc_test/changefeed_auto_stop/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_auto_stop/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/05/30 15:02:14.130 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/05/30 15:02:14.130 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/05/30 15:02:14.157 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/05/30 15:02:14.165 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/05/30 15:02:14.234 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/05/30 15:02:14.444 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/05/30 15:02:14.534 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] + set +x [2024/05/30 15:02:14.629 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] [2024/05/30 15:02:14.730 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] [2024/05/30 15:02:14.739 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] + check + services=($(cat $WORK_DIR/test_proxy.log | xargs -L1 dirname | sort | uniq)) ++ cat /tmp/tidb_cdc_test/http_proxies/test_proxy.log ++ xargs -L1 dirname ++ sort ++ uniq + service_type_count=40 + echo 'captured services: ' captured services: + echo . /cdcpb.ChangeData /grpc.health.v1.Health /pdpb.PD /resource_manager.ResourceManager /tikvpb.Tikv '[2024/05' '[metadata=MD{:authority=[10.233.106.86:20160],' 'user-agent=[grpc-go/1.64.0],' 'grpc-accept-encoding=[gzip],' 'features=[stream-multiplexing],' 'content-type=[application' '[metadata=MD{:authority=[10.233.106.86:2379],' 'grpc-accept-encoding=[gzip],' 'user-agent=[grpc-go/1.64.0],' 'content-type=[application' '[metadata=MD{:authority=[10.233.106.86:2379],' 'user-agent=[grpc-go/1.64.0],' 'grpc-accept-encoding=[gzip],' 'content-type=[application' '[metadata=MD{content-type=[application/grpc],' ':authority=[10.233.106.86:2379],' 'grpc-accept-encoding=[gzip],' 'user-agent=[grpc-go' '[metadata=MD{content-type=[application/grpc],' ':authority=[10.233.106.86:2379],' 'user-agent=[grpc-go' '[metadata=MD{grpc-accept-encoding=[gzip],' 'content-type=[application/grpc],' ':authority=[10.233.106.86:20160],' 'features=[stream-multiplexing],' 'user-agent=[grpc-go' '[metadata=MD{grpc-accept-encoding=[gzip],' 'content-type=[application/grpc],' ':authority=[10.233.106.86:2379],' 'user-agent=[grpc-go' '[method=/cdcpb.ChangeData' '[method=/pdpb.PD' '[method=/resource_manager.ResourceManager' '[stack=main.intercept\n\t/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/http_proxies/run-proxy.go:56\ngithub.com/bradleyjkemp/grpc-tools/grpc-proxy.recoverWrapper.func1\n\t/go/pkg/mod/github.com/bradleyjkemp/grpc-tools@v0.2.5/grpc-proxy/config.go:52\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.64.0/server.go:1680\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.64.0/server.go:1800\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.64.0' . /cdcpb.ChangeData /grpc.health.v1.Health /pdpb.PD /resource_manager.ResourceManager /tikvpb.Tikv [2024/05 [metadata=MD{:authority=[10.233.106.86:20160], user-agent=[grpc-go/1.64.0], grpc-accept-encoding=[gzip], features=[stream-multiplexing], content-type=[application [metadata=MD{:authority=[10.233.106.86:2379], grpc-accept-encoding=[gzip], user-agent=[grpc-go/1.64.0], content-type=[application [metadata=MD{:authority=[10.233.106.86:2379], user-agent=[grpc-go/1.64.0], grpc-accept-encoding=[gzip], content-type=[application [metadata=MD{content-type=[application/grpc], :authority=[10.233.106.86:2379], grpc-accept-encoding=[gzip], user-agent=[grpc-go [metadata=MD{content-type=[application/grpc], :authority=[10.233.106.86:2379], user-agent=[grpc-go [metadata=MD{grpc-accept-encoding=[gzip], content-type=[application/grpc], :authority=[10.233.106.86:20160], features=[stream-multiplexing], user-agent=[grpc-go [metadata=MD{grpc-accept-encoding=[gzip], content-type=[application/grpc], :authority=[10.233.106.86:2379], user-agent=[grpc-go [method=/cdcpb.ChangeData [method=/pdpb.PD [method=/resource_manager.ResourceManager [stack=main.intercept\n\t/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/http_proxies/run-proxy.go:56\ngithub.com/bradleyjkemp/grpc-tools/grpc-proxy.recoverWrapper.func1\n\t/go/pkg/mod/github.com/bradleyjkemp/grpc-tools@v0.2.5/grpc-proxy/config.go:52\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.64.0/server.go:1680\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.64.0/server.go:1800\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.64.0 + '[' 40 -ge 2 ']' + check_logs /tmp/tidb_cdc_test/http_proxies ++ date + echo '[Thu May 30 15:02:14 CST 2024] <<<<<< run test case http_proxies success! >>>>>>' [Thu May 30 15:02:14 CST 2024] <<<<<< run test case http_proxies success! >>>>>> + stop_tidb_cluster failed proxying s2c rpc error: code = Unavailable desc = client disconnected failed proxying s2c rpc error: code = Unavailable desc = client disconnected failed proxying s2c rpc error: code = Canceled desc = context canceled failed proxying s2c rpc error: code = Canceled desc = context canceled =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/open_protocol_handle_key_only/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:02:14 CST 2024] <<<<<< run test case open_protocol_handle_key_only success! >>>>>> [2024/05/30 15:02:15.049 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs7d94c819_0db2_43be_8c55_44e144d46925"] [2024/05/30 15:02:15.037 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark18000] [2024/05/30 15:02:15.037 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark18000] [2024/05/30 15:02:15.046 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark17000] [2024/05/30 15:02:15.133 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark17000] [ts=450115019265015813] [2024/05/30 15:02:15.133 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark17000] [2024/05/30 15:02:15.133 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=450115019265015813] [2024/05/30 15:02:15.135 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:02:15.139 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:02:15.144 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:02:15.148 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:02:15.143 +08:00] [INFO] [main.go:178] ["73 insert success: 1500"] [2024/05/30 15:02:15.242 +08:00] [INFO] [main.go:178] ["73 insert success: 1500"] [2024/05/30 15:02:15.331 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLsd729bcd7_abdc_4b6f_b2a8_45ccb48a5e48"] [2024/05/30 15:02:15.332 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs08a86f3c_03c6_41bd_8ef6_3c4512f73e9a"] [2024/05/30 15:02:15.334 +08:00] [INFO] [main.go:178] ["72 insert success: 1600"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:02:15.458 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:02:15.472 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kafka_big_messages/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:02:14 CST 2024] <<<<<< run test case kafka_big_messages success! >>>>>> [2024/05/30 15:02:15.632 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLsee4743d1_8c94_4d46_b2ac_19edc8621ac6"] [2024/05/30 15:02:15.729 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs5a75af88_4f78_46ef_8c10_574a7c6fb94a"] [2024/05/30 15:02:15.834 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLsf458bd95_b576_49d8_87bb_b61bf1638e15"] [2024/05/30 15:02:15.836 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs315185db_7e93_4aa9_a703_8e0c797ae845"] check diff successfully Resume changefeed with checkpointTs1 450115015511638017 table test.finish_mark not exists for 57-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } table test.table3 not exists for 1-th check, retry later Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc already exists) table processor_delay.t38 exists table processor_delay.t39 not exists for 1-th check, retry later [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [2024/05/30 15:02:16.125 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:02:16.127 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:02:16.131 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:02:16.327 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [Pipeline] // withCredentials [2024/05/30 15:02:16.326 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0016cad20/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:02:16.335 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0016e23c0/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [Pipeline] } [2024/05/30 15:02:16.541 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:02:16.620 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [2024/05/30 15:02:16.732 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:02:16.821 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:02:16.844 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:02:16.923 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:02:16.926 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:02:16.926 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:02:16.935 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:02:16.938 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [Pipeline] // container [Pipeline] } [2024/05/30 15:02:17.037 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark18000] [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:02:17 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/30808058-2b48-40cb-96eb-f5a4cd29dbb4 {"id":"30808058-2b48-40cb-96eb-f5a4cd29dbb4","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84df2bf51 30808058-2b48-40cb-96eb-f5a4cd29dbb4 /tidb/cdc/default/default/upstream/7374684358651933970 {"id":7374684358651933970,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/30808058-2b48-40cb-96eb-f5a4cd29dbb4 {"id":"30808058-2b48-40cb-96eb-f5a4cd29dbb4","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84df2bf51 30808058-2b48-40cb-96eb-f5a4cd29dbb4 /tidb/cdc/default/default/upstream/7374684358651933970 {"id":7374684358651933970,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/30808058-2b48-40cb-96eb-f5a4cd29dbb4 {"id":"30808058-2b48-40cb-96eb-f5a4cd29dbb4","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84df2bf51 30808058-2b48-40cb-96eb-f5a4cd29dbb4 /tidb/cdc/default/default/upstream/7374684358651933970 {"id":7374684358651933970,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Thu May 30 15:02:17 CST 2024] <<<<<< START cdc server in changefeed_auto_stop case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/pipeline/ProcessorSyncResolvedError=1*return(true);github.com/pingcap/tiflow/cdc/processor/ProcessorUpdatePositionDelaying=sleep(1000)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_auto_stop.1099010992.out server --log-file /tmp/tidb_cdc_test/changefeed_auto_stop/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_auto_stop/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] // podTemplate ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] } < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:02:17 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/62a1737f-249d-41ad-8fac-3463e7c296aa {UpstreamID:7374683973007108072 Namespace:default ID:62a1737f-249d-41ad-8fac-3463e7c296aa SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 15:00:34.544368661 +0800 CST StartTs:450114993367023619 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00390fb90 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114993393238022} {CheckpointTs:450115009149403152 MinTableBarrierTs:450115017562652677 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450115009149403152, checkpointTs: 450115009149403152, state: Prepared span: {table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}, resolvedTs: 450115009149403152, checkpointTs: 450115009149403152, state: Prepared span: {table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}, resolvedTs: 450115009149403152, checkpointTs: 450115009149403152, state: Prepared span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115009149403152, checkpointTs: 450115009149403152, state: Prepared span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 450115009149403152, checkpointTs: 450115009149403152, state: Prepared span: {table_id:111,start_key:7480000000000000ff6f5f720000000000fa,end_key:7480000000000000ff6f5f730000000000fa}, resolvedTs: 450115009149403152, checkpointTs: 450115009149403152, state: Prepared span: {table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}, resolvedTs: 450115009149403152, checkpointTs: 450115009149403152, state: Prepared span: {table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}, resolvedTs: 450115009149403152, checkpointTs: 450115009149403152, state: Prepared *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0d8fa6ec-f678-46d2-9511-f88ec7a365fd {"id":"0d8fa6ec-f678-46d2-9511-f88ec7a365fd","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c999384 0d8fa6ec-f678-46d2-9511-f88ec7a365fd /tidb/cdc/default/default/changefeed/info/62a1737f-249d-41ad-8fac-3463e7c296aa {"upstream-id":7374683973007108072,"namespace":"default","changefeed-id":"62a1737f-249d-41ad-8fac-3463e7c296aa","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T15:00:34.544368661+08:00","start-ts":450114993367023619,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_storage_file/redo","use-file-backend":false,"compression":"lz4","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114993393238022} /tidb/cdc/default/default/changefeed/status/62a1737f-249d-41ad-8fac-3463e7c296aa {"checkpoint-ts":450115009149403152,"min-table-barrier-ts":450115018899587074,"admin-job-type":0} /tidb/cdc/default/default/task/position/0d8fa6ec-f678-46d2-9511-f88ec7a365fd/62a1737f-249d-41ad-8fac-3463e7c296aa {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683973007108072 {"id":7374683973007108072,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/62a1737f-249d-41ad-8fac-3463e7c296aa {UpstreamID:7374683973007108072 Namespace:default ID:62a1737f-249d-41ad-8fac-3463e7c296aa SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 15:00:34.544368661 +0800 CST StartTs:450114993367023619 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00390fb90 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114993393238022} {CheckpointTs:450115009149403152 MinTableBarrierTs:450115017562652677 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450115009149403152, checkpointTs: 450115009149403152, state: Prepared span: {table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}, resolvedTs: 450115009149403152, checkpointTs: 450115009149403152, state: Prepared span: {table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}, resolvedTs: 450115009149403152, checkpointTs: 450115009149403152, state: Prepared span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115009149403152, checkpointTs: 450115009149403152, state: Prepared span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 450115009149403152, checkpointTs: 450115009149403152, state: Prepared span: {table_id:111,start_key:7480000000000000ff6f5f720000000000fa,end_key:7480000000000000ff6f5f730000000000fa}, resolvedTs: 450115009149403152, checkpointTs: 450115009149403152, state: Prepared span: {table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}, resolvedTs: 450115009149403152, checkpointTs: 450115009149403152, state: Prepared span: {table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}, resolvedTs: 450115009149403152, checkpointTs: 450115009149403152, state: Prepared *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0d8fa6ec-f678-46d2-9511-f88ec7a365fd {"id":"0d8fa6ec-f678-46d2-9511-f88ec7a365fd","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c999384 0d8fa6ec-f678-46d2-9511-f88ec7a365fd /tidb/cdc/default/default/changefeed/info/62a1737f-249d-41ad-8fac-3463e7c296aa {"upstream-id":7374683973007108072,"namespace":"default","changefeed-id":"62a1737f-249d-41ad-8fac-3463e7c296aa","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T15:00:34.544368661+08:00","start-ts":450114993367023619,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_storage_file/redo","use-file-backend":false,"compression":"lz4","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114993393238022} /tidb/cdc/default/default/changefeed/status/62a1737f-249d-41ad-8fac-3463e7c296aa {"checkpoint-ts":450115009149403152,"min-table-barrier-ts":450115018899587074,"admin-job-type":0} /tidb/cdc/default/default/task/position/0d8fa6ec-f678-46d2-9511-f88ec7a365fd/62a1737f-249d-41ad-8fac-3463e7c296aa {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683973007108072 {"id":7374683973007108072,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/62a1737f-249d-41ad-8fac-3463e7c296aa {UpstreamID:7374683973007108072 Namespace:default ID:62a1737f-249d-41ad-8fac-3463e7c296aa SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 15:00:34.544368661 +0800 CST StartTs:450114993367023619 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00390fb90 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114993393238022} {CheckpointTs:450115009149403152 MinTableBarrierTs:450115017562652677 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450115009149403152, checkpointTs: 450115009149403152, state: Prepared span: {table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}, resolvedTs: 450115009149403152, checkpointTs: 450115009149403152, state: Prepared span: {table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}, resolvedTs: 450115009149403152, checkpointTs: 450115009149403152, state: Prepared span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115009149403152, checkpointTs: 450115009149403152, state: Prepared span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 450115009149403152, checkpointTs: 450115009149403152, state: Prepared span: {table_id:111,start_key:7480000000000000ff6f5f720000000000fa,end_key:7480000000000000ff6f5f730000000000fa}, resolvedTs: 450115009149403152, checkpointTs: 450115009149403152, state: Prepared span: {table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}, resolvedTs: 450115009149403152, checkpointTs: 450115009149403152, state: Prepared span: {table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}, resolvedTs: 450115009149403152, checkpointTs: 450115009149403152, state: Prepared *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0d8fa6ec-f678-46d2-9511-f88ec7a365fd {"id":"0d8fa6ec-f678-46d2-9511-f88ec7a365fd","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c999384 0d8fa6ec-f678-46d2-9511-f88ec7a365fd /tidb/cdc/default/default/changefeed/info/62a1737f-249d-41ad-8fac-3463e7c296aa {"upstream-id":7374683973007108072,"namespace":"default","changefeed-id":"62a1737f-249d-41ad-8fac-3463e7c296aa","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T15:00:34.544368661+08:00","start-ts":450114993367023619,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_storage_file/redo","use-file-backend":false,"compression":"lz4","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114993393238022} /tidb/cdc/default/default/changefeed/status/62a1737f-249d-41ad-8fac-3463e7c296aa {"checkpoint-ts":450115009149403152,"min-table-barrier-ts":450115018899587074,"admin-job-type":0} /tidb/cdc/default/default/task/position/0d8fa6ec-f678-46d2-9511-f88ec7a365fd/62a1737f-249d-41ad-8fac-3463e7c296aa {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683973007108072 {"id":7374683973007108072,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x [2024/05/30 15:02:17.234 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:02:17.235 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:02:17.325 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:02:17.447 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs0cd95857_88d7_4916_a54a_48d66d9a9cfa"] [2024/05/30 15:02:17.521 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kafka_compression/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:02:17 CST 2024] <<<<<< run test case kafka_compression success! >>>>>> [Pipeline] // withEnv [Pipeline] } [2024/05/30 15:02:17.752 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] table test.finish_mark not exists for 58-th check, retry later [Pipeline] // stage [Pipeline] } [2024/05/30 15:02:17.832 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:02:17.934 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:02:18.021 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:02:18.031 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:02:18.037 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:02:18.038 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:02:18.039 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:02:18.042 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:02:18.044 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] table test.table3 not exists for 2-th check, retry later table processor_delay.t39 exists table processor_delay.t40 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/canal_json_claim_check/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:02:18 CST 2024] <<<<<< run test case canal_json_claim_check success! >>>>>> [2024/05/30 15:02:18.229 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:02:18.231 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:02:18.321 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:02:18.624 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:02:18.629 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:02:18.634 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:02:18.841 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:18.941 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:19.018 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:02:19.122 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:02:19.136 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:02:19.137 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:02:19.143 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:02:19.229 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:02:19.232 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:02:19.233 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:02:19.236 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:19.244 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:19.329 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:19.435 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:19.445 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:02:19.522 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:02:19.057 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark18000] [2024/05/30 15:02:19.228 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark18000] [ts=450115020292096005] [2024/05/30 15:02:19.228 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark18000] [2024/05/30 15:02:19.228 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=450115020292096005] [2024/05/30 15:02:19.238 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:02:19.242 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:02:19.247 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:02:19.333 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] table test.finish_mark not exists for 59-th check, retry later [2024/05/30 15:02:19.831 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:19.836 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark19000] [2024/05/30 15:02:19.837 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark19000] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2139db840015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:12625, start at 2024-05-30 15:02:16.560599485 +0800 CST m=+5.224170746 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:16.567 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:16.545 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:16.545 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2139db840015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:12625, start at 2024-05-30 15:02:16.560599485 +0800 CST m=+5.224170746 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:16.567 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:16.545 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:16.545 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2139de8c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:12714, start at 2024-05-30 15:02:16.754646476 +0800 CST m=+5.368187167 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:16.762 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:16.739 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:16.739 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/savepoint/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/savepoint/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table processor_delay.t40 exists table processor_delay.t41 not exists for 1-th check, retry later [2024/05/30 15:02:19.581 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0016cad20/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:02:19.649 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0016e23c0/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] skip some SQLs in tidb v4.0.x [2024/05/30 15:02:20.030 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:20.129 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:20.148 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:20.234 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:20.236 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:20.340 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:20.342 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] table test.table3 not exists for 3-th check, retry later table common_1.v1 exists table common_1.recover_and_insert not exists for 1-th check, retry later [2024/05/30 15:02:20.429 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:20.431 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:20.435 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:20.444 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:20.530 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:20.534 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:20.535 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:02:19.647 +08:00] [WARN] [diff.go:556] ["checksum is not equal"] [table=`test`.`ntest`] [where="((TRUE) AND TRUE)"] [sourceChecksum=0] [targetChecksum=1549695251] [getSourceChecksumCost=1.89757ms] [getTargetChecksumCost=3.43329ms] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:02:20 GMT < Content-Length: 867 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/048993cf-3550-41ad-94b2-6645c2d95cd9 {"id":"048993cf-3550-41ad-94b2-6645c2d95cd9","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/30808058-2b48-40cb-96eb-f5a4cd29dbb4 {"id":"30808058-2b48-40cb-96eb-f5a4cd29dbb4","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84df2bf51 30808058-2b48-40cb-96eb-f5a4cd29dbb4 /tidb/cdc/default/__cdc_meta__/owner/22318fc84df2bf82 048993cf-3550-41ad-94b2-6645c2d95cd9 /tidb/cdc/default/default/upstream/7374684358651933970 {"id":7374684358651933970,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/048993cf-3550-41ad-94b2-6645c2d95cd9 {"id":"048993cf-3550-41ad-94b2-6645c2d95cd9","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/30808058-2b48-40cb-96eb-f5a4cd29dbb4 {"id":"30808058-2b48-40cb-96eb-f5a4cd29dbb4","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84df2bf51 30808058-2b48-40cb-96eb-f5a4cd29dbb4 /tidb/cdc/default/__cdc_meta__/owner/22318fc84df2bf82 048993cf-3550-41ad-94b2-6645c2d95cd9 /tidb/cdc/default/default/upstream/7374684358651933970 {"id":7374684358651933970,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/048993cf-3550-41ad-94b2-6645c2d95cd9 {"id":"048993cf-3550-41ad-94b2-6645c2d95cd9","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/30808058-2b48-40cb-96eb-f5a4cd29dbb4 {"id":"30808058-2b48-40cb-96eb-f5a4cd29dbb4","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84df2bf51 30808058-2b48-40cb-96eb-f5a4cd29dbb4 /tidb/cdc/default/__cdc_meta__/owner/22318fc84df2bf82 048993cf-3550-41ad-94b2-6645c2d95cd9 /tidb/cdc/default/default/upstream/7374684358651933970 {"id":7374684358651933970,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_changefeed_state http://127.0.0.1:2379 72324d65-7da4-48c5-b11d-3f38aacb26e7 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=72324d65-7da4-48c5-b11d-3f38aacb26e7 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 72324d65-7da4-48c5-b11d-3f38aacb26e7 -s + info='{ "upstream_id": 7374684358651933970, "namespace": "default", "id": "72324d65-7da4-48c5-b11d-3f38aacb26e7", "state": "normal", "checkpoint_tso": 450115019196858369, "checkpoint_time": "2024-05-30 15:02:12.947", "error": null }' + echo '{ "upstream_id": 7374684358651933970, "namespace": "default", "id": "72324d65-7da4-48c5-b11d-3f38aacb26e7", "state": "normal", "checkpoint_tso": 450115019196858369, "checkpoint_time": "2024-05-30 15:02:12.947", "error": null }' { "upstream_id": 7374684358651933970, "namespace": "default", "id": "72324d65-7da4-48c5-b11d-3f38aacb26e7", "state": "normal", "checkpoint_tso": 450115019196858369, "checkpoint_time": "2024-05-30 15:02:12.947", "error": null } ++ echo '{' '"upstream_id":' 7374684358651933970, '"namespace":' '"default",' '"id":' '"72324d65-7da4-48c5-b11d-3f38aacb26e7",' '"state":' '"normal",' '"checkpoint_tso":' 450115019196858369, '"checkpoint_time":' '"2024-05-30' '15:02:12.947",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374684358651933970, '"namespace":' '"default",' '"id":' '"72324d65-7da4-48c5-b11d-3f38aacb26e7",' '"state":' '"normal",' '"checkpoint_tso":' 450115019196858369, '"checkpoint_time":' '"2024-05-30' '15:02:12.947",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully table changefeed_auto_stop_1.usertable not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kafka_messages/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:02:20 CST 2024] <<<<<< run test case kafka_messages success! >>>>>> [2024/05/30 15:02:20.644 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:02:20.828 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:02:21.018 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:21.235 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:02:21.332 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] table test.finish_mark not exists for 60-th check, retry later [2024/05/30 15:02:21.429 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:21.429 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:21.620 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:21.626 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.cli.14031.out cli tso query --pd=http://127.0.0.1:2379 table processor_delay.t41 exists table processor_delay.t42 not exists for 1-th check, retry later [2024/05/30 15:02:21.719 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:21.731 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:21.731 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:21.735 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:02:21.748 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:02:21.750 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:21.751 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:02:21.930 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:22.123 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/open_protocol_claim_check/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:02:21 CST 2024] <<<<<< run test case open_protocol_claim_check success! >>>>>> [2024/05/30 15:02:21.837 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark19000] [2024/05/30 15:02:21.854 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark19000] [ts=450115021493239813] [2024/05/30 15:02:21.854 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark19000] [2024/05/30 15:02:21.854 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=450115021493239813] [2024/05/30 15:02:21.856 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:02:21.860 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:02:21.866 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:02:21.873 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:02:22.231 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] table test.table3 exists table test.table4 not exists for 1-th check, retry later table common_1.recover_and_insert not exists for 2-th check, retry later [2024/05/30 15:02:22.631 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:02:22.635 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:22.723 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:02:22.733 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:02:22.858 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:02:22.864 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] table changefeed_auto_stop_1.usertable exists table changefeed_auto_stop_2.usertable not exists for 1-th check, retry later [2024/05/30 15:02:23.119 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:02:23.122 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:23.123 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:23.124 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:02:23.124 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:02:23.128 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] table test.finish_mark not exists for 61-th check, retry later [2024/05/30 15:02:23.227 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:23.326 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:23.432 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:02:23.529 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] table processor_delay.t42 exists table processor_delay.t43 not exists for 1-th check, retry later + set +x + tso='450115021581844485 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115021581844485 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 15:02:23 CST 2024] <<<<<< START cdc server in savepoint case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.1405614058.out server --log-file /tmp/tidb_cdc_test/savepoint/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/savepoint/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/05/30 15:02:16.334 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001588780/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:02:16.334 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0019263c0/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:02:19.225 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0015881e0/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:02:23.841 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:02:23.935 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kafka_sink_error_resume/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:02:23 CST 2024] <<<<<< run test case kafka_sink_error_resume success! >>>>>> [2024/05/30 15:02:24.021 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:02:24.024 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:02:24.141 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] table test.table4 exists check diff failed 1-th time, retry later [2024/05/30 15:02:24.227 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:24.326 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:02:24.333 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:02:24.419 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:24.422 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:24.426 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:24.431 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:02:24.618 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:02:24.632 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:02:24.637 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:02:24.638 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:02:24.568 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0019261e0/127.0.0.1:2379] [attempt=0] [error="rpc error: code = NotFound desc = etcdserver: requested lease not found"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/canal_json_storage_basic/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:02:24 CST 2024] <<<<<< run test case canal_json_storage_basic success! >>>>>> [2024/05/30 15:02:24.869 +08:00] [INFO] [case.go:556] ["one upstream workload finished"] [round=20001] [2024/05/30 15:02:24.869 +08:00] [INFO] [case.go:603] ["one downstream exit due to receive done"] [2024/05/30 15:02:24.869 +08:00] [INFO] [case.go:556] ["one upstream workload finished"] [round=20002] [2024/05/30 15:02:24.869 +08:00] [INFO] [case.go:603] ["one downstream exit due to receive done"] [2024/05/30 15:02:24.869 +08:00] [INFO] [case.go:556] ["one upstream workload finished"] [round=20003] [2024/05/30 15:02:24.869 +08:00] [INFO] [case.go:603] ["one downstream exit due to receive done"] [2024/05/30 15:02:24.870 +08:00] [INFO] [case.go:556] ["one upstream workload finished"] [round=20004] [2024/05/30 15:02:24.870 +08:00] [INFO] [case.go:603] ["one downstream exit due to receive done"] table common_1.recover_and_insert not exists for 3-th check, retry later table changefeed_auto_stop_2.usertable exists table changefeed_auto_stop_3.usertable not exists for 1-th check, retry later [2024/05/30 15:02:24.956 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark20000] [2024/05/30 15:02:24.956 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark20000] [2024/05/30 15:02:24.956 +08:00] [INFO] [case.go:556] ["one upstream workload finished"] [round=20004] [2024/05/30 15:02:24.407 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0016ca1e0/127.0.0.1:2379] [attempt=0] [error="rpc error: code = NotFound desc = etcdserver: requested lease not found"] [2024/05/30 15:02:25.027 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:02:25.045 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:02:25.137 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:02:25.148 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:25.341 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:02:25.421 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:02:25.432 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:02:25.443 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:02:25.521 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:02:25.521 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:02:25.526 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:25.527 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:02:25.634 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:25.727 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:25.728 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:02:25.728 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] table test.finish_mark exists check diff successfully table processor_delay.t43 exists table processor_delay.t44 not exists for 1-th check, retry later [2024/05/30 15:02:26.242 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:02:26.246 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:02:26.336 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] wait process cdc.test exit for 1-th time... [2024/05/30 15:02:26.426 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:02:26.535 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:26.549 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:26.620 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] check diff successfully table common_1.recover_and_insert exists table common_1.finish_mark not exists for 1-th check, retry later [2024/05/30 15:02:26.629 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:26.635 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:26.639 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:26.640 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:26.642 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:02:26.740 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:02:26.742 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:02:26.742 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:26.747 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] table changefeed_auto_stop_3.usertable exists table changefeed_auto_stop_4.usertable not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:02:26 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dacf03b6-9d35-4ee2-b441-1e3d842adea0 {"id":"dacf03b6-9d35-4ee2-b441-1e3d842adea0","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84e4e6fc4 dacf03b6-9d35-4ee2-b441-1e3d842adea0 /tidb/cdc/default/default/upstream/7374684450826674080 {"id":7374684450826674080,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dacf03b6-9d35-4ee2-b441-1e3d842adea0 {"id":"dacf03b6-9d35-4ee2-b441-1e3d842adea0","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84e4e6fc4 dacf03b6-9d35-4ee2-b441-1e3d842adea0 /tidb/cdc/default/default/upstream/7374684450826674080 {"id":7374684450826674080,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dacf03b6-9d35-4ee2-b441-1e3d842adea0 {"id":"dacf03b6-9d35-4ee2-b441-1e3d842adea0","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84e4e6fc4 dacf03b6-9d35-4ee2-b441-1e3d842adea0 /tidb/cdc/default/default/upstream/7374684450826674080 {"id":7374684450826674080,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.cli.14101.out cli changefeed create --start-ts=450115021581844485 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ [2024/05/30 15:02:26.957 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark20000] wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... Create changefeed successfully! ID: fbae7cca-b679-4bd8-a903-8ca7ea462ad7 Info: {"upstream_id":7374684450826674080,"namespace":"default","id":"fbae7cca-b679-4bd8-a903-8ca7ea462ad7","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T15:02:27.150181348+08:00","start_ts":450115021581844485,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115021581844485,"checkpoint_ts":450115021581844485,"checkpoint_time":"2024-05-30 15:02:22.045"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... wait process cdc.test exit for 3-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/mq_sink_lost_callback/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:02:26 CST 2024] <<<<<< run test case mq_sink_lost_callback success! >>>>>> [2024/05/30 15:02:27.428 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:27.434 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] wait process cdc.test exit for 2-th time... [2024/05/30 15:02:27.446 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:27.552 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:27.647 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 15:02:27 CST 2024] <<<<<< START cdc server in changefeed_resume_with_checkpoint_ts case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/InjectChangefeedFastFailError=1*return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_resume_with_checkpoint_ts.85288530.out server --log-file /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/cdc_data --cluster-id default --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:02:27 CST 2024] <<<<<< run test case tidb_mysql_test success! >>>>>> [2024/05/30 15:02:27.733 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:02:27.740 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:02:27.834 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:02:27.835 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:02:27.835 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:02:27.839 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:27.842 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:02:27.844 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:02:27.845 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:02:27.929 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:27.937 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/canal_json_storage_partition_table/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:02:27 CST 2024] <<<<<< run test case canal_json_storage_partition_table success! >>>>>> table processor_delay.t44 exists table processor_delay.t45 not exists for 1-th check, retry later [2024/05/30 15:02:28.333 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:02:28.419 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:02:28.444 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] + set +x table common_1.finish_mark not exists for 2-th check, retry later [2024/05/30 15:02:28.728 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] table changefeed_auto_stop_4.usertable exists check diff failed 1-th time, retry later [2024/05/30 15:02:28.820 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:28.828 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:02:28.920 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:02:28.932 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:28.938 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:02:28.943 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:28.943 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:28.945 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:29.035 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:02:29.146 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:02:29.148 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:02:29.238 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:02:28.971 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark20000] [2024/05/30 15:02:28.986 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark20000] [ts=450115022921924613] [2024/05/30 15:02:28.987 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark20000] [2024/05/30 15:02:28.987 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=450115022921924613] [2024/05/30 15:02:28.989 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:02:28.992 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:02:28.996 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/05/30 15:02:29.000 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/05/30 15:02:29.000 +08:00] [INFO] [case.go:603] ["one downstream exit due to receive done"] [2024/05/30 15:02:29.000 +08:00] [INFO] [case.go:615] ["bank test finished"] [valid=20] [tries=20] [ratio=1] + cleanup_process cdc.test wait process cdc.test exit for 1-th time... [2024/05/30 15:02:29.625 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:02:29.748 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/05/30 15:02:29.838 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:02:29.846 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] table processor_delay.t45 exists table processor_delay.t46 not exists for 1-th check, retry later [2024/05/30 15:02:29.858 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:02:29.860 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:02:29.927 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:30.021 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/05/30 15:02:30.028 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/05/30 15:02:30.055 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] wait process cdc.test exit for 2-th time... [2024/05/30 15:02:30.219 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:02:30.347 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] table savepoint.finish_mark not exists for 1-th check, retry later table common_1.finish_mark not exists for 3-th check, retry later [2024/05/30 15:02:30.425 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/05/30 15:02:30.435 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/05/30 15:02:30.523 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/05/30 15:02:30.529 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/05/30 15:02:30.536 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] wait process cdc.test exit for 3-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/mq_sink_dispatcher/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:02:30 CST 2024] <<<<<< run test case mq_sink_dispatcher success! >>>>>> [2024/05/30 15:02:30.728 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/05/30 15:02:30.735 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/05/30 15:02:30.851 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/05/30 15:02:30.942 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/05/30 15:02:31.064 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/05/30 15:02:31.123 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/05/30 15:02:31.127 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:02:30 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/21aa005e-174d-4c8a-bf97-f43112de6d74 {"id":"21aa005e-174d-4c8a-bf97-f43112de6d74","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318fc84dbce9d0 21aa005e-174d-4c8a-bf97-f43112de6d74 /tidb/cdc/default/default/changefeed/info/8d7af8b6-a3c4-4273-be1a-cd3f333bcc81 {"upstream-id":7374684287058025977,"namespace":"default","changefeed-id":"8d7af8b6-a3c4-4273-be1a-cd3f333bcc81","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T15:01:58.7889031+08:00","start-ts":450115015511638017,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"failed","error":{"time":"2024-05-30T15:02:28.317446202+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrStartTsBeforeGC","message":"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"},"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115023232040961} /tidb/cdc/default/default/changefeed/status/8d7af8b6-a3c4-4273-be1a-cd3f333bcc81 {"checkpoint-ts":450115017477718046,"min-table-barrier-ts":450115017517039629,"admin-job-type":1} /tidb/cdc/default/default/upstream/7374684287058025977 {"id":7374684287058025977,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/21aa005e-174d-4c8a-bf97-f43112de6d74 {"id":"21aa005e-174d-4c8a-bf97-f43112de6d74","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318fc84dbce9d0 21aa005e-174d-4c8a-bf97-f43112de6d74 /tidb/cdc/default/default/changefeed/info/8d7af8b6-a3c4-4273-be1a-cd3f333bcc81 {"upstream-id":7374684287058025977,"namespace":"default","changefeed-id":"8d7af8b6-a3c4-4273-be1a-cd3f333bcc81","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T15:01:58.7889031+08:00","start-ts":450115015511638017,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"failed","error":{"time":"2024-05-30T15:02:28.317446202+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrStartTsBeforeGC","message":"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"},"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115023232040961} /tidb/cdc/default/default/changefeed/status/8d7af8b6-a3c4-4273-be1a-cd3f333bcc81 {"checkpoint-ts":450115017477718046,"min-table-barrier-ts":450115017517039629,"admin-job-type":1} /tidb/cdc/default/default/upstream/7374684287058025977 {"id":7374684287058025977,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/21aa005e-174d-4c8a-bf97-f43112de6d74 {"id":"21aa005e-174d-4c8a-bf97-f43112de6d74","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318fc84dbce9d0 21aa005e-174d-4c8a-bf97-f43112de6d74 /tidb/cdc/default/default/changefeed/info/8d7af8b6-a3c4-4273-be1a-cd3f333bcc81 {"upstream-id":7374684287058025977,"namespace":"default","changefeed-id":"8d7af8b6-a3c4-4273-be1a-cd3f333bcc81","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T15:01:58.7889031+08:00","start-ts":450115015511638017,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"failed","error":{"time":"2024-05-30T15:02:28.317446202+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrStartTsBeforeGC","message":"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"},"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115023232040961} /tidb/cdc/default/default/changefeed/status/8d7af8b6-a3c4-4273-be1a-cd3f333bcc81 {"checkpoint-ts":450115017477718046,"min-table-barrier-ts":450115017517039629,"admin-job-type":1} /tidb/cdc/default/default/upstream/7374684287058025977 {"id":7374684287058025977,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_changefeed_state http://127.0.0.1:2379 8d7af8b6-a3c4-4273-be1a-cd3f333bcc81 failed ErrStartTsBeforeGC + endpoints=http://127.0.0.1:2379 + changefeed_id=8d7af8b6-a3c4-4273-be1a-cd3f333bcc81 + expected_state=failed + error_msg=ErrStartTsBeforeGC + tls_dir=ErrStartTsBeforeGC + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 8d7af8b6-a3c4-4273-be1a-cd3f333bcc81 -s + info='{ "upstream_id": 7374684287058025977, "namespace": "default", "id": "8d7af8b6-a3c4-4273-be1a-cd3f333bcc81", "state": "failed", "checkpoint_tso": 450115017477718046, "checkpoint_time": "2024-05-30 15:02:06.389", "error": { "time": "2024-05-30T15:02:28.317446202+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError" } }' + echo '{ "upstream_id": 7374684287058025977, "namespace": "default", "id": "8d7af8b6-a3c4-4273-be1a-cd3f333bcc81", "state": "failed", "checkpoint_tso": 450115017477718046, "checkpoint_time": "2024-05-30 15:02:06.389", "error": { "time": "2024-05-30T15:02:28.317446202+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError" } }' { "upstream_id": 7374684287058025977, "namespace": "default", "id": "8d7af8b6-a3c4-4273-be1a-cd3f333bcc81", "state": "failed", "checkpoint_tso": 450115017477718046, "checkpoint_time": "2024-05-30 15:02:06.389", "error": { "time": "2024-05-30T15:02:28.317446202+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError" } } ++ jq -r .state ++ echo '{' '"upstream_id":' 7374684287058025977, '"namespace":' '"default",' '"id":' '"8d7af8b6-a3c4-4273-be1a-cd3f333bcc81",' '"state":' '"failed",' '"checkpoint_tso":' 450115017477718046, '"checkpoint_time":' '"2024-05-30' '15:02:06.389",' '"error":' '{' '"time":' '"2024-05-30T15:02:28.317446202+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"' '}' '}' cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit ++ date + echo '[Thu May 30 15:02:31 CST 2024] <<<<<< run test case bank success! >>>>>>' [Thu May 30 15:02:31 CST 2024] <<<<<< run test case bank success! >>>>>> + stop_tidb_cluster [2024/05/30 15:02:30.519 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001588780/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:02:30.536 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0019263c0/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] check diff successfully =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/multi_tables_ddl/run.sh using Sink-Type: mysql... <<================= * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:24927; Connection refused * Closing connection 0 [2024/05/30 15:02:31.158 +08:00] [INFO] [main.go:178] ["73 insert success: 1500"] [2024/05/30 15:02:31.242 +08:00] [INFO] [main.go:178] ["73 insert success: 1500"] [2024/05/30 15:02:31.329 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs0bca0e74_a74c_4504_8fa4_bf817f0ea3a7"] [2024/05/30 15:02:31.335 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs6b84453d_3079_49e6_a3c7_f1f87bff1810"] [2024/05/30 15:02:31.342 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ jq -r .error.message ++ echo '{' '"upstream_id":' 7374684287058025977, '"namespace":' '"default",' '"id":' '"8d7af8b6-a3c4-4273-be1a-cd3f333bcc81",' '"state":' '"failed",' '"checkpoint_tso":' 450115017477718046, '"checkpoint_time":' '"2024-05-30' '15:02:06.389",' '"error":' '{' '"time":' '"2024-05-30T15:02:28.317446202+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"' '}' '}' + message='[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError' + [[ ! [CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError =~ ErrStartTsBeforeGC ]] run task successfully wait process cdc.test exit for 1-th time... [2024/05/30 15:02:31.621 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs2e7224f7_34c4_49c3_a92a_e76492704571"] [2024/05/30 15:02:31.628 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] check_changefeed_state http://127.0.0.1:2379 8d7af8b6-a3c4-4273-be1a-cd3f333bcc81 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=8d7af8b6-a3c4-4273-be1a-cd3f333bcc81 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 8d7af8b6-a3c4-4273-be1a-cd3f333bcc81 -s [2024/05/30 15:02:30.507 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0016cad20/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:02:30.508 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0016e23c0/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:02:31.838 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] table processor_delay.t46 exists table processor_delay.t47 not exists for 1-th check, retry later + info='{ "upstream_id": 7374684287058025977, "namespace": "default", "id": "8d7af8b6-a3c4-4273-be1a-cd3f333bcc81", "state": "normal", "checkpoint_tso": 450115024031580164, "checkpoint_time": "2024-05-30 15:02:31.390", "error": null }' + echo '{ "upstream_id": 7374684287058025977, "namespace": "default", "id": "8d7af8b6-a3c4-4273-be1a-cd3f333bcc81", "state": "normal", "checkpoint_tso": 450115024031580164, "checkpoint_time": "2024-05-30 15:02:31.390", "error": null }' { "upstream_id": 7374684287058025977, "namespace": "default", "id": "8d7af8b6-a3c4-4273-be1a-cd3f333bcc81", "state": "normal", "checkpoint_tso": 450115024031580164, "checkpoint_time": "2024-05-30 15:02:31.390", "error": null } ++ echo '{' '"upstream_id":' 7374684287058025977, '"namespace":' '"default",' '"id":' '"8d7af8b6-a3c4-4273-be1a-cd3f333bcc81",' '"state":' '"normal",' '"checkpoint_tso":' 450115024031580164, '"checkpoint_time":' '"2024-05-30' '15:02:31.390",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374684287058025977, '"namespace":' '"default",' '"id":' '"8d7af8b6-a3c4-4273-be1a-cd3f333bcc81",' '"state":' '"normal",' '"checkpoint_tso":' 450115024031580164, '"checkpoint_time":' '"2024-05-30' '15:02:31.390",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully You are running an older version of MinIO released 3 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend [2024/05/30 15:02:31.919 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/05/30 15:02:31.943 +08:00] [INFO] [main.go:178] ["73 insert success: 1600"] [2024/05/30 15:02:32.038 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs74f4c4af_1a9a_48c2_a54e_a9fe764e9eaf"] wait process cdc.test exit for 2-th time... {"level":"warn","ts":"2024-05-30T15:02:32.185188+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000e8ae00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} {"level":"warn","ts":"2024-05-30T15:02:32.185517+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001123a40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} table savepoint.finish_mark exists check diff successfully [2024/05/30 15:02:32.235 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:02:32.236 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:02:32.237 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:02:32.339 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:02:32.344 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:02:32.348 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLsc16f67a1_9c04_48a5_86e1_a437b21c0223"] table common_1.finish_mark exists table test1.table1 not exists for 1-th check, retry later Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide [2024/05/30 15:02:32.530 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] wait process cdc.test exit for 3-th time... check diff successfully wait process cdc.test exit for 1-th time... [2024/05/30 15:02:32.735 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:02:32.819 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] + stop_proxy + kill 8560 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/sequence/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:02:31 CST 2024] <<<<<< run test case sequence success! >>>>>> [2024/05/30 15:02:18.850 +08:00] [INFO] [pd_service_discovery.go:631] ["[pd] switch leader"] [new-leader=http://127.0.0.1:2379] [old-leader=] [2024/05/30 15:02:18.850 +08:00] [INFO] [pd_service_discovery.go:197] ["[pd] init cluster id"] [cluster-id=7374683768182076119] [2024/05/30 15:02:18.850 +08:00] [INFO] [client.go:600] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2024/05/30 15:02:18.850 +08:00] [INFO] [tso_client.go:231] ["[tso] switch dc tso global allocator serving address"] [dc-location=global] [new-address=http://127.0.0.1:2379] [2024/05/30 15:02:18.851 +08:00] [INFO] [tso_dispatcher.go:323] ["[tso] tso dispatcher created"] [dc-location=global] [2024/05/30 15:02:18.851 +08:00] [INFO] [client.go:648] ["[pd] service mode changed"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2024/05/30 15:02:18.852 +08:00] [INFO] [pd_service_discovery.go:631] ["[pd] switch leader"] [new-leader=http://127.0.0.1:2379] [old-leader=] [2024/05/30 15:02:18.852 +08:00] [INFO] [pd_service_discovery.go:197] ["[pd] init cluster id"] [cluster-id=7374683768182076119] [2024/05/30 15:02:18.852 +08:00] [INFO] [client.go:600] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2024/05/30 15:02:18.852 +08:00] [INFO] [tso_client.go:231] ["[tso] switch dc tso global allocator serving address"] [dc-location=global] [new-address=http://127.0.0.1:2379] [2024/05/30 15:02:18.853 +08:00] [INFO] [tso_dispatcher.go:323] ["[tso] tso dispatcher created"] [dc-location=global] [2024/05/30 15:02:18.853 +08:00] [INFO] [client.go:648] ["[pd] service mode changed"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2024/05/30 15:02:18.853 +08:00] [INFO] [tikv_driver.go:200] ["using API V1."] [2024/05/30 15:02:18.854 +08:00] [INFO] [main.go:180] ["genLock started"] [2024/05/30 15:02:18.855 +08:00] [INFO] [region_cache.go:2683] ["change store resolve state"] [store=3] [addr=127.0.0.1:20162] [from=unresolved] [to=resolved] [liveness-state=reachable] [2024/05/30 15:02:28.873 +08:00] [INFO] [main.go:196] ["genLock done"] [2024/05/30 15:02:28.874 +08:00] [INFO] [pd_service_discovery.go:248] ["[pd] exit member loop due to context canceled"] [2024/05/30 15:02:28.874 +08:00] [INFO] [tso_dispatcher.go:186] ["exit tso requests cancel loop"] [2024/05/30 15:02:28.874 +08:00] [INFO] [resource_manager_client.go:271] ["[resource manager] exit resource token dispatcher"] [2024/05/30 15:02:28.874 +08:00] [INFO] [tso_dispatcher.go:410] ["[tso] stop fetching the pending tso requests due to context canceled"] [dc-location=global] [2024/05/30 15:02:28.874 +08:00] [INFO] [tso_client.go:134] ["closing tso client"] [2024/05/30 15:02:28.874 +08:00] [INFO] [tso_dispatcher.go:344] ["[tso] exit tso dispatcher"] [dc-location=global] [2024/05/30 15:02:28.874 +08:00] [INFO] [tso_dispatcher.go:240] ["exit tso dispatcher loop"] [2024/05/30 15:02:28.874 +08:00] [INFO] [tso_client.go:139] ["close tso client"] [2024/05/30 15:02:28.874 +08:00] [INFO] [tso_client.go:150] ["tso client is closed"] [2024/05/30 15:02:28.874 +08:00] [INFO] [pd_service_discovery.go:294] ["[pd] close pd service discovery client"] [2024/05/30 15:02:33.150 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:02:33.165 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:02:33.179 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:02:32.833 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001588780/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:02:33.221 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0019263c0/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:02:32.972 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0016cad20/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:02:33.018 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0016e23c0/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:02:33 CST 2024] <<<<<< run test case changefeed_auto_stop success! >>>>>> wait process cdc.test exit for 1-th time... * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:24927 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 17D432332BB59372 < X-Xss-Protection: 1; mode=block < Date: Thu, 30 May 2024 07:02:33 GMT < { [data not shown] * Connection #0 to host 127.0.0.1 left intact [2024/05/30 15:02:33.261 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:02:33.264 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:02:33.333 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:02:33.441 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] wait process cdc.test exit for 2-th time... Bucket 's3://logbucket/' created The 1 times to try to start tidb cluster... [2024/05/30 15:02:33.527 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] wait process cdc.test exit for 2-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kafka_column_selector/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:02:33 CST 2024] <<<<<< run test case kafka_column_selector success! >>>>>> [2024/05/30 15:02:33.731 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:02:33.822 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLsb6fd894e_147a_43fa_a2be_1b149b096da7"] [2024/05/30 15:02:33.828 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:02:34.118 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs0da44911_8061_4666_aaa5_0cf2e52703f1"] \033[0;36m<<< Run all test success >>>\033[0m table test1.table1 not exists for 2-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 15:02:34 CST 2024] <<<<<< run test case common_1 success! >>>>>> [2024/05/30 15:02:34.225 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:02:34.247 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:02:34.250 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:02:34.336 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:02:34.345 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:02:34.419 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:02:34.528 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:02:34.529 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:02:34.623 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLse02acee1_a77a_41cb_8ae2_205ff77709e7"] [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc already exists) [Pipeline] // cache [2024/05/30 15:02:34.733 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:02:34.822 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:02:34.920 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:02:34.923 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [2024/05/30 15:02:35.237 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [2024/05/30 15:02:35.338 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [2024/05/30 15:02:35.348 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001588780/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 15:02:35.518 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:35.520 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:35.538 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:35.541 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:35.628 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:35.633 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:35.634 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:02:35.636 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:02:35.731 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:02:35.737 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:02:35.626 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0013541e0/127.0.0.1:2479] [attempt=0] [error="rpc error: code = NotFound desc = etcdserver: requested lease not found"] [Pipeline] // node [Pipeline] } wait process cdc.test exit for 3-th time... [Pipeline] // podTemplate [Pipeline] } [2024/05/30 15:02:35.822 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:35.836 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] table processor_delay.t47 exists table processor_delay.t48 exists table processor_delay.t49 not exists for 1-th check, retry later [Pipeline] // withEnv [Pipeline] } table test1.table1 exists table test1.table2 not exists for 1-th check, retry later [Pipeline] // stage [2024/05/30 15:02:36.025 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:36.029 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [Pipeline] } table test.t2 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kafka_column_selector_avro/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:02:36 CST 2024] <<<<<< run test case kafka_column_selector_avro success! >>>>>> cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:02:36 CST 2024] <<<<<< run test case savepoint success! >>>>>> [2024/05/30 15:02:36.433 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:02:36.719 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] start tidb cluster in /tmp/tidb_cdc_test/multi_tables_ddl Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... [2024/05/30 15:02:36.826 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:36.847 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:36.848 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:36.855 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:36.950 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:36.950 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:36.951 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:37.022 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:02:37.032 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:37.045 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:37.118 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:02:37.122 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:02:37.335 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:37.422 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:02:37.740 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/ddl_reentrant/run.sh using Sink-Type: mysql... <<================= ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) The 1 times to try to start tidb cluster... table processor_delay.t49 exists table processor_delay.t50 not exists for 1-th check, retry later [2024/05/30 15:02:37.944 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] table test.t2 not exists for 2-th check, retry later [2024/05/30 15:02:38.123 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:02:38.131 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:02:38.149 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:38.224 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:02:38.226 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:02:38.234 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:02:38.238 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:02:38.322 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:02:38.322 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:38.322 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:02:38.424 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:38.435 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:02:38.532 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] table test1.table2 exists check diff successfully [2024/05/30 15:02:38.636 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:38.935 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:39.324 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:39.421 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:02:39.421 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:02:39.429 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:02:39.434 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:02:39.443 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:39.447 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:39.449 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:02:39.526 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:39.538 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:39.542 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:39.550 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:02:39.623 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:39.641 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:39.652 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:39.820 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/lossy_ddl/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:02:39 CST 2024] <<<<<< run test case lossy_ddl success! >>>>>> wait process cdc.test exit for 1-th time... table test.t2 not exists for 3-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release start tidb cluster in /tmp/tidb_cdc_test/ddl_reentrant Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... [2024/05/30 15:02:40.127 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:40.243 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:02:40.322 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:02:40.327 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:02:40.337 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:02:40.343 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:02:40.344 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:40.423 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:02:40.432 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] table processor_delay.t50 exists check diff failed 1-th time, retry later Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... wait process cdc.test exit for 2-th time... [2024/05/30 15:02:40.436 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:02:40.444 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:02:40.529 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:40.622 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:40.641 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:02:40.727 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:02:40.827 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kill_owner_with_ddl/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 15:02:41 CST 2024] <<<<<< run test case changefeed_resume_with_checkpoint_ts success! >>>>>> [2024/05/30 15:02:41.419 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:02:41.441 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:02:41.445 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:02:41.522 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:02:41.543 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:02:41.550 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:02:41.630 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:02:41.641 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:02:41.646 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:41.723 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:41.734 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:41.746 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] table test.t2 not exists for 4-th check, retry later [2024/05/30 15:02:41.826 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:02:41.846 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:41.850 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:02:41.934 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [2024/05/30 15:02:42.420 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:42.429 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:42.431 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:02:42.433 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:42.528 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:42.531 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:02:42.543 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:02:42.627 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:42.638 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:02:42.648 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:02:42.732 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:02:42.738 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:42.747 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:42.825 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:02:42.836 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] check diff successfully [2024/05/30 15:02:42.927 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/storage_csv_update/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:02:42 CST 2024] <<<<<< run test case storage_csv_update success! >>>>>> \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc already exists) [2024/05/30 15:02:43.352 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [Pipeline] // cache [Pipeline] } wait process cdc.test exit for 1-th time... [Pipeline] // dir [2024/05/30 15:02:43.420 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:43.420 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:02:43.436 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:02:43.438 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:02:43.526 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:02:43.539 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:02:43.551 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } table test.t2 not exists for 5-th check, retry later [2024/05/30 15:02:43.643 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:43.658 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:43.737 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:43.741 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:02:43.821 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:02:43.824 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:02:43.830 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [Pipeline] // stage [Pipeline] } wait process cdc.test exit for 2-th time... [Pipeline] // container [Pipeline] } [2024/05/30 15:02:43.937 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } start tidb cluster in /tmp/tidb_cdc_test/kill_owner_with_ddl Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... [Pipeline] // podTemplate [Pipeline] } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // withEnv cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 15:02:44 CST 2024] <<<<<< run test case processor_etcd_worker_delay success! >>>>>> [Pipeline] } [2024/05/30 15:02:44.440 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:02:44.525 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:02:44.530 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:44.530 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:02:44.535 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:02:44.718 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:44.730 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:02:44.737 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [Pipeline] // stage [Pipeline] } Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:02:44.836 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:02:44.840 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:02:44.933 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:02:45.020 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:45.036 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:45.133 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:45.137 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:02:45.238 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc already exists) table test.t2 not exists for 6-th check, retry later [2024/05/30 15:02:45.542 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:02:45.544 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:02:45.621 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/05/30 15:02:45.720 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:02:45.733 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/05/30 15:02:45.734 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:45.823 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:02:45.829 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [2024/05/30 15:02:45.852 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:02:45.918 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:02:45.924 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/05/30 15:02:45.954 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [2024/05/30 15:02:46.243 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:46.320 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/05/30 15:02:46.346 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [Pipeline] // stage [Pipeline] } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/run.sh using Sink-Type: mysql... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status + source /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/synced_status + CDC_BINARY=cdc.test + SINK_TYPE=mysql + CDC_COUNT=3 + DB_COUNT=4 + trap stop_tidb_cluster EXIT + run_normal_case_and_unavailable_pd conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // container \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } [2024/05/30 15:02:46.450 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:02:46.532 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:02:46.536 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [Pipeline] // withEnv [Pipeline] } [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc already exists) [2024/05/30 15:02:46.628 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/05/30 15:02:46.653 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:02:46.726 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:46.735 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/05/30 15:02:46.827 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [Pipeline] // node ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // cache [Pipeline] } [Pipeline] } [2024/05/30 15:02:47.018 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/05/30 15:02:47.021 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [Pipeline] // podTemplate [Pipeline] // dir [Pipeline] } [Pipeline] } [Pipeline] // withEnv [Pipeline] // withCredentials [2024/05/30 15:02:47.145 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/05/30 15:02:47.229 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:02:47.234 +08:00] [INFO] [main.go:178] ["73 insert success: 1500"] [Pipeline] } [Pipeline] } [Pipeline] // stage [Pipeline] // timeout [Pipeline] } [Pipeline] } [2024/05/30 15:02:47.420 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/05/30 15:02:47.521 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/05/30 15:02:47.540 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:02:47.629 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [2024/05/30 15:02:47.751 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/05/30 15:02:47.867 +08:00] [INFO] [main.go:178] ["73 insert success: 1500"] [2024/05/30 15:02:47.868 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [2024/05/30 15:02:47.984 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/05/30 15:02:48.034 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/05/30 15:02:48.048 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs86bf973b_b78a_4e61_bf34_a85567ec2a04"] [2024/05/30 15:02:48.122 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] [2024/05/30 15:02:48.148 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [Pipeline] // podTemplate [Pipeline] } table test.t2 not exists for 7-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // withEnv [Pipeline] } [2024/05/30 15:02:48.158 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/05/30 15:02:48.257 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [Pipeline] // stage [Pipeline] } [2024/05/30 15:02:48.456 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/05/30 15:02:48.557 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/05/30 15:02:48.572 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:02:48.642 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:02:48.649 +08:00] [INFO] [main.go:178] ["73 insert success: 1500"] [2024/05/30 15:02:48.664 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:02:48.854 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs3763e83e_359c_4684_84de_41cc16e353d8"] start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... [2024/05/30 15:02:48.940 +08:00] [INFO] [main.go:178] ["73 insert success: 1500"] [2024/05/30 15:02:48.956 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:02:49.064 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:02:49.131 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] [2024/05/30 15:02:49.273 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:02:49.327 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:02:49.341 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLsa44136ab_be17_49e4_965a_43e514f446e3"] [2024/05/30 15:02:49.432 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs91e6ca13_291d_4599_acd4_35a552d380f8"] [2024/05/30 15:02:49.446 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:02:49.539 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [2024/05/30 15:02:49.820 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs8db8fc0f_892b_4752_bab3_f18aa1f218ab"] [2024/05/30 15:02:49.847 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:02:49.926 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:02:49.931 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] table test.t2 not exists for 8-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:02:50.039 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:02:50.129 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:02:50.139 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:50.236 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:02:50.254 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213be200000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-bcfm9, pid:6412, start at 2024-05-30 15:02:49.73864914 +0800 CST m=+5.262828018 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:49.746 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:49.728 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:49.728 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213be200000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-bcfm9, pid:6412, start at 2024-05-30 15:02:49.73864914 +0800 CST m=+5.262828018 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:49.746 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:49.728 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:49.728 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213be29c0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-bcfm9, pid:6471, start at 2024-05-30 15:02:49.800110809 +0800 CST m=+5.270502142 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:49.807 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:49.767 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:49.767 +0800 All versions after safe point can be accessed. (DO NOT EDIT) [2024/05/30 15:02:50.547 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/05/30 15:02:50.558 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/05/30 15:02:50.633 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:02:50.721 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:02:50.724 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_reentrant/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_reentrant/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_reentrant/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_reentrant/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_reentrant/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2024/05/30 15:02:50.825 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:02:50.838 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/05/30 15:02:50.854 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:50.938 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:02:50.964 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... [2024/05/30 15:02:51.226 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:02:51.248 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/05/30 15:02:51.319 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:51.323 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:51.352 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:02:51.420 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:02:51.527 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:02:51.625 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:02:51.665 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/05/30 15:02:51.744 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:02:51.930 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/05/30 15:02:51.957 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] table test.t2 not exists for 9-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:02:52.034 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:52.049 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:52.050 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:52.051 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:52.125 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:52.218 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:02:52.332 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:52.427 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:52.630 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/05/30 15:02:52.721 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/05/30 15:02:52.848 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:02:52.922 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:02:52.922 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:52.929 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:52.964 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:53.053 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:02:53.133 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/05/30 15:02:53.159 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] \033[0;36m<<< Run all test success >>>\033[0m skip some DDLs in tidb v4.0.x [Thu May 30 15:02:52 CST 2024] <<<<<< START cdc server in ddl_reentrant case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_reentrant.77517753.out server --log-file /tmp/tidb_cdc_test/ddl_reentrant/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_reentrant/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [2024/05/30 15:02:53.443 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/05/30 15:02:53.449 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [2024/05/30 15:02:53.646 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:02:53.650 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:02:53.651 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:02:53.655 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:53.727 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/05/30 15:02:53.770 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:02:53.834 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [Pipeline] // stage [Pipeline] } table test.t2 not exists for 10-th check, retry later [Pipeline] // container [Pipeline] } [2024/05/30 15:02:53.920 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:54.039 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [2024/05/30 15:02:54.133 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/05/30 15:02:54.247 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:02:54.253 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:02:54.266 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:54.328 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:02:54.346 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [Pipeline] // podTemplate Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... [Pipeline] } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/sink_hang/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] // withEnv [Pipeline] } [2024/05/30 15:02:54.434 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:54.459 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:54.559 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213c0e880014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-js3jc, pid:30666, start at 2024-05-30 15:02:52.622988691 +0800 CST m=+10.663799602 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:52.640 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:52.631 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:52.631 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213c0e880014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-js3jc, pid:30666, start at 2024-05-30 15:02:52.622988691 +0800 CST m=+10.663799602 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:52.640 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:52.631 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:52.631 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213c0ee00014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-js3jc, pid:30737, start at 2024-05-30 15:02:52.639413013 +0800 CST m=+10.606494363 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:52.649 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:52.650 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:52.650 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_tables_ddl/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_tables_ddl/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/multi_tables_ddl/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_tables_ddl/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/multi_tables_ddl/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // stage [Pipeline] } [2024/05/30 15:02:54.744 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/05/30 15:02:54.940 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/05/30 15:02:55.142 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:02:55.146 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:02:55.229 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:02:55.239 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:55.241 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/05/30 15:02:55.259 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:02:55.357 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/05/30 15:02:55.441 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:55.623 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.t2 not exists for 11-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:02:55 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/83cd4770-a515-4657-9c9e-4e29cfb19000 {"id":"83cd4770-a515-4657-9c9e-4e29cfb19000","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ed3f4b7 83cd4770-a515-4657-9c9e-4e29cfb19000 /tidb/cdc/default/default/upstream/7374684597706537036 {"id":7374684597706537036,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/83cd4770-a515-4657-9c9e-4e29cfb19000 {"id":"83cd4770-a515-4657-9c9e-4e29cfb19000","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ed3f4b7 83cd4770-a515-4657-9c9e-4e29cfb19000 /tidb/cdc/default/default/upstream/7374684597706537036 {"id":7374684597706537036,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/83cd4770-a515-4657-9c9e-4e29cfb19000 {"id":"83cd4770-a515-4657-9c9e-4e29cfb19000","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ed3f4b7 83cd4770-a515-4657-9c9e-4e29cfb19000 /tidb/cdc/default/default/upstream/7374684597706537036 {"id":7374684597706537036,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [2024/05/30 15:02:55.838 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] ------------------------------------------ test ddl create database ddl_reentrant, is_reentrant: false restored_sql: CREATE DATABASE `ddl_reentrant` [Thu May 30 15:02:56 CST 2024] <<<<<< START cdc server in multi_tables_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_tables_ddl.3200532007.out server --log-file /tmp/tidb_cdc_test/multi_tables_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_tables_ddl/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check_ts_forward baafe3a1-81aa-4051-9d66-9bafc75e9067 [2024/05/30 15:02:56.736 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:56.740 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:02:56.764 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:56.830 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:02:56.843 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/05/30 15:02:56.859 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:02:56.941 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:56.958 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:02:57.068 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:02:57.438 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/05/30 15:02:57.632 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:02:57.637 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:57.649 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] start tidb cluster in /tmp/tidb_cdc_test/sink_hang Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... [2024/05/30 15:02:57.664 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:57.678 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:57.739 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:02:57.765 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/05/30 15:02:57.842 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/05/30 15:02:58.131 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] table test.t2 exists check diff successfully [2024/05/30 15:02:58.134 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:02:58.139 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:02:58.152 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:02:58.181 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:02:58.240 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:58.337 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/05/30 15:02:58.352 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/05/30 15:02:58.580 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] wait process cdc.test exit for 1-th time... changefeed is working normally rts: 450115030464069600->450115030883762200 checkpoint: 450115030464069600->450115030883762200 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql true [2024/05/30 15:02:57.756 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="CREATE DATABASE `ddl_reentrant`"] [duration=80.436587ms] [namespace=default] [changefeed=baafe3a1-81aa-4051-9d66-9bafc75e9067] run task successfully Verifying downstream PD is started... [2024/05/30 15:02:58.651 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/05/30 15:02:58.752 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/05/30 15:02:58.753 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:02:59.043 +08:00] [INFO] [main.go:812] ["testMultiDDLs take %v1m20.254808558s"] wait process cdc.test exit for 2-th time... resume changefeed baafe3a1-81aa-4051-9d66-9bafc75e9067 from 450115030503391264 check_ts_forward baafe3a1-81aa-4051-9d66-9bafc75e9067 [2024/05/30 15:02:59.132 +08:00] [INFO] [main.go:74] ["DefaultValue integration tests take 1m20.343572338s"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213c4c6c0005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-0vnff, pid:6362, start at 2024-05-30 15:02:56.546317486 +0800 CST m=+5.462062865 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:56.554 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:56.539 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:56.539 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213c4c6c0005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-0vnff, pid:6362, start at 2024-05-30 15:02:56.546317486 +0800 CST m=+5.462062865 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:56.554 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:56.539 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:56.539 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213c4cac0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-0vnff, pid:6414, start at 2024-05-30 15:02:56.602029026 +0800 CST m=+5.451979226 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:56.609 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:56.604 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:56.604 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/kill_owner_with_ddl/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/kill_owner_with_ddl/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/kill_owner_with_ddl/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/kill_owner_with_ddl/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/kill_owner_with_ddl/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table mark.finish_mark_1 not exists for 1-th check, retry later wait process cdc.test exit for 3-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:02:59 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a6288b74-2663-4497-9327-4db1297b1b39 {"id":"a6288b74-2663-4497-9327-4db1297b1b39","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ec62ec6 a6288b74-2663-4497-9327-4db1297b1b39 /tidb/cdc/default/default/upstream/7374684582254680420 {"id":7374684582254680420,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a6288b74-2663-4497-9327-4db1297b1b39 {"id":"a6288b74-2663-4497-9327-4db1297b1b39","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ec62ec6 a6288b74-2663-4497-9327-4db1297b1b39 /tidb/cdc/default/default/upstream/7374684582254680420 {"id":7374684582254680420,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a6288b74-2663-4497-9327-4db1297b1b39 {"id":"a6288b74-2663-4497-9327-4db1297b1b39","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ec62ec6 a6288b74-2663-4497-9327-4db1297b1b39 /tidb/cdc/default/default/upstream/7374684582254680420 {"id":7374684582254680420,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: test-normal Info: {"upstream_id":7374684582254680420,"namespace":"default","id":"test-normal","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T15:02:59.801449402+08:00","start_ts":450115030608248833,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["multi_tables_ddl_test.t1","multi_tables_ddl_test.t2","multi_tables_ddl_test.t3","multi_tables_ddl_test.t4","multi_tables_ddl_test.t1_7","multi_tables_ddl_test.t2_7","multi_tables_ddl_test.finish_mark"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115030608248833,"checkpoint_ts":450115030608248833,"checkpoint_time":"2024-05-30 15:02:56.478"} Create changefeed successfully! ID: test-error-1 Info: {"upstream_id":7374684582254680420,"namespace":"default","id":"test-error-1","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T15:03:00.055943392+08:00","start_ts":450115030608248833,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["multi_tables_ddl_test.t5","multi_tables_ddl_test.t6","multi_tables_ddl_test.t7","multi_tables_ddl_test.t8"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115030608248833,"checkpoint_ts":450115030608248833,"checkpoint_time":"2024-05-30 15:02:56.478"} cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:02:59 CST 2024] <<<<<< run test case resolve_lock success! >>>>>> Create changefeed successfully! ID: test-error-2 Info: {"upstream_id":7374684582254680420,"namespace":"default","id":"test-error-2","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T15:03:00.252010763+08:00","start_ts":450115030608248833,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["multi_tables_ddl_test.t9","multi_tables_ddl_test.t10"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115030608248833,"checkpoint_ts":450115030608248833,"checkpoint_time":"2024-05-30 15:02:56.478"} VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213c79340018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:14997, start at 2024-05-30 15:02:59.435627993 +0800 CST m=+4.993450158 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:59.441 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:59.405 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:59.405 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213c79340018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:14997, start at 2024-05-30 15:02:59.435627993 +0800 CST m=+4.993450158 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:59.441 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:59.405 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:59.405 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213c7e600014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:15064, start at 2024-05-30 15:02:59.768120125 +0800 CST m=+5.245335007 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:59.774 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:59.736 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:59.736 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release changefeed is working normally rts: 450115031145906200->450115031669932000 checkpoint: 450115030503391200->450115031669932000 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql false [2024/05/30 15:02:59.634 +08:00] [INFO] [mysql_ddl_sink.go:119] ["Execute DDL failed, but error can be ignored"] [startTs=450115030503391259] [ddl="CREATE DATABASE `ddl_reentrant`"] [namespace=default] [changefeed=baafe3a1-81aa-4051-9d66-9bafc75e9067] [error="Error 1007 (HY000): Can't create database 'ddl_reentrant'; database exists"] run task successfully ------------------------------------------ test ddl create table ddl_reentrant.t3 (a int primary key, b int) partition by range(a) (partition p0 values less than (1000), partition p1 values less than (2000)), is_reentrant: false restored_sql: CREATE TABLE `ddl_reentrant`.`t3` (`a` INT PRIMARY KEY,`b` INT) PARTITION BY RANGE (`a`) (PARTITION `p0` VALUES LESS THAN (1000),PARTITION `p1` VALUES LESS THAN (2000)) check_ts_forward baafe3a1-81aa-4051-9d66-9bafc75e9067 [Thu May 30 15:03:01 CST 2024] <<<<<< START cdc server in kill_owner_with_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kill_owner_with_ddl.76457647.out server --log-file /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table mark.finish_mark_1 exists table mark.finish_mark_2 not exists for 1-th check, retry later + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.16428.out cli tso query --pd=http://127.0.0.1:2379 Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) run task failed 1-th time, retry later table mark.finish_mark_2 not exists for 2-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:03:04 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f95c8b76-09d1-460d-9a0c-8fb1e240a5e1 {"id":"f95c8b76-09d1-460d-9a0c-8fb1e240a5e1","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ee1f7b7 f95c8b76-09d1-460d-9a0c-8fb1e240a5e1 /tidb/cdc/default/default/upstream/7374684617514448989 {"id":7374684617514448989,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f95c8b76-09d1-460d-9a0c-8fb1e240a5e1 {"id":"f95c8b76-09d1-460d-9a0c-8fb1e240a5e1","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ee1f7b7 f95c8b76-09d1-460d-9a0c-8fb1e240a5e1 /tidb/cdc/default/default/upstream/7374684617514448989 {"id":7374684617514448989,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f95c8b76-09d1-460d-9a0c-8fb1e240a5e1 {"id":"f95c8b76-09d1-460d-9a0c-8fb1e240a5e1","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ee1f7b7 f95c8b76-09d1-460d-9a0c-8fb1e240a5e1 /tidb/cdc/default/default/upstream/7374684617514448989 {"id":7374684617514448989,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + set +x + tso='450115032345477121 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115032345477121 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=450115032345477121 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Thu May 30 15:03:04 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.1645816460.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Create changefeed successfully! ID: 743c206c-db17-4f29-b103-9e56a2826b2d Info: {"upstream_id":7374684617514448989,"namespace":"default","id":"743c206c-db17-4f29-b103-9e56a2826b2d","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T15:03:04.811090645+08:00","start_ts":450115032747606021,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115032747606021,"checkpoint_ts":450115032747606021,"checkpoint_time":"2024-05-30 15:03:04.639"} ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_2 not exists for 3-th check, retry later table kill_owner_with_ddl.t1 not exists for 1-th check, retry later check_ts_forward baafe3a1-81aa-4051-9d66-9bafc75e9067 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check_redo_resolved_ts 62a1737f-249d-41ad-8fac-3463e7c296aa 450115032963874819 file:///tmp/tidb_cdc_test/consistent_replicate_storage_file/redo /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc_data/redo/62a1737f-249d-41ad-8fac-3463e7c296aa/meta global resolved ts 450115020395118594 not forward to 450115032963874819 run task failed 1-th time, retry later check_redo_resolved_ts 62a1737f-249d-41ad-8fac-3463e7c296aa 450115032963874819 file:///tmp/tidb_cdc_test/consistent_replicate_storage_file/redo /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc_data/redo/62a1737f-249d-41ad-8fac-3463e7c296aa/meta global resolved ts 450115020395118594 not forward to 450115032963874819 run task failed 2-th time, retry later table mark.finish_mark_2 not exists for 4-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:03:07 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dd961e70-ee68-4717-af8a-18d0d08af62c {"id":"dd961e70-ee68-4717-af8a-18d0d08af62c","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ef4c5bd dd961e70-ee68-4717-af8a-18d0d08af62c /tidb/cdc/default/default/upstream/7374684640737950414 {"id":7374684640737950414,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dd961e70-ee68-4717-af8a-18d0d08af62c {"id":"dd961e70-ee68-4717-af8a-18d0d08af62c","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ef4c5bd dd961e70-ee68-4717-af8a-18d0d08af62c /tidb/cdc/default/default/upstream/7374684640737950414 {"id":7374684640737950414,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dd961e70-ee68-4717-af8a-18d0d08af62c {"id":"dd961e70-ee68-4717-af8a-18d0d08af62c","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ef4c5bd dd961e70-ee68-4717-af8a-18d0d08af62c /tidb/cdc/default/default/upstream/7374684640737950414 {"id":7374684640737950414,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=450115032345477121 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.16503.out cli changefeed create --start-ts=450115032345477121 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml table kill_owner_with_ddl.t1 exists changefeed is working normally rts: 450115032980652000->450115033242796000 checkpoint: 450115032980652000->450115033242796000 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql true [2024/05/30 15:03:01.661 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="CREATE TABLE `ddl_reentrant`.`t3` (`a` INT PRIMARY KEY,`b` INT) PARTITION BY RANGE (`a`) (PARTITION `p0` VALUES LESS THAN (1000),PARTITION `p1` VALUES LESS THAN (2000))"] [duration=84.766945ms] [namespace=default] [changefeed=baafe3a1-81aa-4051-9d66-9bafc75e9067] run task successfully Create changefeed successfully! ID: test-1 Info: {"upstream_id":7374684640737950414,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T15:03:08.019151+08:00","start_ts":450115032345477121,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115032345477121,"checkpoint_ts":450115032345477121,"checkpoint_time":"2024-05-30 15:03:03.105"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... resume changefeed baafe3a1-81aa-4051-9d66-9bafc75e9067 from 450115031840325634 check_ts_forward baafe3a1-81aa-4051-9d66-9bafc75e9067 wait process 7650 exit for 1-th time... wait process 7650 exit for 2-th time... wait process 7650 exit for 3-th time... + set +x ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2681 0 --:--:-- --:--:-- --:--:-- 2662 100 221 100 221 0 0 2678 0 --:--:-- --:--:-- --:--:-- 2662 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-30 15:03:03.105","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-05-30 15:03:09.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:03:03.105","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-30' '15:03:09.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:03:03.105","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-30' '15:03:09.000","info":"Data' syncing is 'finished"}' ++ jq -r .sink_checkpoint_ts + sink_checkpoint_ts='2024-05-30 15:03:03.105' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:03:03.105","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-30' '15:03:09.000","info":"Data' syncing is 'finished"}' ++ jq -r .puller_resolved_ts + puller_resolved_ts='1970-01-01 08:00:00.000' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:03:03.105","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-30' '15:03:09.000","info":"Data' syncing is 'finished"}' ++ jq -r .last_synced_ts VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213d0600000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-p8f0f, pid:7106, start at 2024-05-30 15:03:08.429815006 +0800 CST m=+5.123082847 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:05:08.436 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:03:08.416 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:53:08.416 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213d0600000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-p8f0f, pid:7106, start at 2024-05-30 15:03:08.429815006 +0800 CST m=+5.123082847 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:05:08.436 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:03:08.416 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:53:08.416 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213d0b740014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-p8f0f, pid:7177, start at 2024-05-30 15:03:08.799377074 +0800 CST m=+5.438641814 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:05:08.806 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:03:08.765 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:53:08.765 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sink_hang/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sink_hang/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + last_synced_ts='1970-01-01 08:00:00.000' + '[' true '!=' true ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' ++ date '+%Y-%m-%d %H:%M:%S' + current='2024-05-30 15:03:09' + echo 'sink_checkpoint_ts is 2024-05-30' 15:03:03.105 sink_checkpoint_ts is 2024-05-30 15:03:03.105 ++ date -d '2024-05-30 15:03:03.105' +%s + checkpoint_timestamp=1717052583 ++ date -d '2024-05-30 15:03:09' +%s + current_timestamp=1717052589 + '[' 6 -gt 300 ']' + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (7650) - No such process wait process 7650 exit for 4-th time... process 7650 already exit check_capture_count http://127.0.0.1:2379 0 run task failed 1-th time, retry later table mark.finish_mark_2 not exists for 5-th check, retry later parse error: Invalid numeric literal at line 1, column 6 run task successfully [Thu May 30 15:03:10 CST 2024] <<<<<< START cdc server in kill_owner_with_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql/MySQLSinkExecDDLDelay=return(true);github.com/pingcap/tiflow/cdc/capture/ownerFlushIntervalInject=return(10)' + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kill_owner_with_ddl.77807782.out server --log-file /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check_redo_resolved_ts 62a1737f-249d-41ad-8fac-3463e7c296aa 450115032963874819 file:///tmp/tidb_cdc_test/consistent_replicate_storage_file/redo /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc_data/redo/62a1737f-249d-41ad-8fac-3463e7c296aa/meta [Thu May 30 15:03:11 CST 2024] <<<<<< START cdc server in sink_hang case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkExecDMLError=2*return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_hang.84888490.out server --log-file /tmp/tidb_cdc_test/sink_hang/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sink_hang/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table mark.finish_mark_2 not exists for 6-th check, retry later table test.t1 exists + sleep 5 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/move_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... global resolved ts 450115020395118594 not forward to 450115032963874819 run task failed 3-th time, retry later check_ts_forward baafe3a1-81aa-4051-9d66-9bafc75e9067 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:03:13 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/743c206c-db17-4f29-b103-9e56a2826b2d {UpstreamID:7374684617514448989 Namespace:default ID:743c206c-db17-4f29-b103-9e56a2826b2d SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-30 15:03:04.811090645 +0800 CST StartTs:450115032747606021 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0035c2750 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115032786927619} {CheckpointTs:450115034674364418 MinTableBarrierTs:450115034936508419 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115034936508418, checkpointTs: 450115034936508418, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4ebed3e9-0911-4dcd-b9ae-954963a24202 {"id":"4ebed3e9-0911-4dcd-b9ae-954963a24202","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ee1f835 4ebed3e9-0911-4dcd-b9ae-954963a24202 /tidb/cdc/default/default/changefeed/info/743c206c-db17-4f29-b103-9e56a2826b2d {"upstream-id":7374684617514448989,"namespace":"default","changefeed-id":"743c206c-db17-4f29-b103-9e56a2826b2d","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T15:03:04.811090645+08:00","start-ts":450115032747606021,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115032786927619} /tidb/cdc/default/default/changefeed/status/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":450115034936508418,"min-table-barrier-ts":450115034936508419,"admin-job-type":0} /tidb/cdc/default/default/task/position/4ebed3e9-0911-4dcd-b9ae-954963a24202/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684617514448989 {"id":7374684617514448989,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/743c206c-db17-4f29-b103-9e56a2826b2d {UpstreamID:7374684617514448989 Namespace:default ID:743c206c-db17-4f29-b103-9e56a2826b2d SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-30 15:03:04.811090645 +0800 CST StartTs:450115032747606021 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0035c2750 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115032786927619} {CheckpointTs:450115034674364418 MinTableBarrierTs:450115034936508419 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115034936508418, checkpointTs: 450115034936508418, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4ebed3e9-0911-4dcd-b9ae-954963a24202 {"id":"4ebed3e9-0911-4dcd-b9ae-954963a24202","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ee1f835 4ebed3e9-0911-4dcd-b9ae-954963a24202 /tidb/cdc/default/default/changefeed/info/743c206c-db17-4f29-b103-9e56a2826b2d {"upstream-id":7374684617514448989,"namespace":"default","changefeed-id":"743c206c-db17-4f29-b103-9e56a2826b2d","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T15:03:04.811090645+08:00","start-ts":450115032747606021,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115032786927619} /tidb/cdc/default/default/changefeed/status/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":450115034936508418,"min-table-barrier-ts":450115034936508419,"admin-job-type":0} /tidb/cdc/default/default/task/position/4ebed3e9-0911-4dcd-b9ae-954963a24202/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684617514448989 {"id":7374684617514448989,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/743c206c-db17-4f29-b103-9e56a2826b2d {UpstreamID:7374684617514448989 Namespace:default ID:743c206c-db17-4f29-b103-9e56a2826b2d SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-30 15:03:04.811090645 +0800 CST StartTs:450115032747606021 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0035c2750 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115032786927619} {CheckpointTs:450115034674364418 MinTableBarrierTs:450115034936508419 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115034936508418, checkpointTs: 450115034936508418, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4ebed3e9-0911-4dcd-b9ae-954963a24202 {"id":"4ebed3e9-0911-4dcd-b9ae-954963a24202","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ee1f835 4ebed3e9-0911-4dcd-b9ae-954963a24202 /tidb/cdc/default/default/changefeed/info/743c206c-db17-4f29-b103-9e56a2826b2d {"upstream-id":7374684617514448989,"namespace":"default","changefeed-id":"743c206c-db17-4f29-b103-9e56a2826b2d","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T15:03:04.811090645+08:00","start-ts":450115032747606021,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115032786927619} /tidb/cdc/default/default/changefeed/status/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":450115034936508418,"min-table-barrier-ts":450115034936508419,"admin-job-type":0} /tidb/cdc/default/default/task/position/4ebed3e9-0911-4dcd-b9ae-954963a24202/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684617514448989 + grep -q 'etcd info' {"id":7374684617514448989,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x check_capture_count http://127.0.0.1:2379 1 run task successfully table mark.finish_mark_2 not exists for 7-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/move_table Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... changefeed is working normally rts: 450115034553516000->450115035078066200 checkpoint: 450115034553516000->450115035078066200 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql false [2024/05/30 15:03:08.686 +08:00] [INFO] [mysql_ddl_sink.go:119] ["Execute DDL failed, but error can be ignored"] [startTs=450115031827218461] [ddl="CREATE TABLE `ddl_reentrant`.`t3` (`a` INT PRIMARY KEY,`b` INT) PARTITION BY RANGE (`a`) (PARTITION `p0` VALUES LESS THAN (1000),PARTITION `p1` VALUES LESS THAN (2000))"] [namespace=default] [changefeed=baafe3a1-81aa-4051-9d66-9bafc75e9067] [error="Error 1050 (42S01): Table 'ddl_reentrant.t3' already exists"] run task successfully ------------------------------------------ test ddl create table ddl_reentrant.t4 (a int primary key, b int), is_reentrant: false restored_sql: CREATE TABLE `ddl_reentrant`.`t4` (`a` INT PRIMARY KEY,`b` INT) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > check_ts_forward baafe3a1-81aa-4051-9d66-9bafc75e9067 < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:03:14 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fee6e5ac-20fe-4d92-9aac-f016ecebeb79 {"id":"fee6e5ac-20fe-4d92-9aac-f016ecebeb79","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f177cba fee6e5ac-20fe-4d92-9aac-f016ecebeb79 /tidb/cdc/default/default/upstream/7374684668304150740 {"id":7374684668304150740,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fee6e5ac-20fe-4d92-9aac-f016ecebeb79 {"id":"fee6e5ac-20fe-4d92-9aac-f016ecebeb79","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f177cba fee6e5ac-20fe-4d92-9aac-f016ecebeb79 /tidb/cdc/default/default/upstream/7374684668304150740 {"id":7374684668304150740,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fee6e5ac-20fe-4d92-9aac-f016ecebeb79 {"id":"fee6e5ac-20fe-4d92-9aac-f016ecebeb79","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f177cba fee6e5ac-20fe-4d92-9aac-f016ecebeb79 /tidb/cdc/default/default/upstream/7374684668304150740 {"id":7374684668304150740,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table multi_tables_ddl_test.t55 exists table multi_tables_ddl_test.t66 exists table multi_tables_ddl_test.t7 exists table multi_tables_ddl_test.t88 exists table multi_tables_ddl_test.finish_mark not exists for 1-th check, retry later table mark.finish_mark_2 not exists for 8-th check, retry later table multi_tables_ddl_test.finish_mark not exists for 2-th check, retry later run task failed 1-th time, retry later check_changefeed_status 127.0.0.1:8300 1e309ec7-2940-4c98-86d7-40ee911ac24f normal last_error null + endpoint=127.0.0.1:8300 + changefeed_id=1e309ec7-2940-4c98-86d7-40ee911ac24f + expected_state=normal + field=last_error + error_pattern=null ++ curl 127.0.0.1:8300/api/v2/changefeeds/1e309ec7-2940-4c98-86d7-40ee911ac24f/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 86 100 86 0 0 831 0 --:--:-- --:--:-- --:--:-- 834 + info='{"state":"normal","resolved_ts":450115035598946339,"checkpoint_ts":450115035598946339}' + echo '{"state":"normal","resolved_ts":450115035598946339,"checkpoint_ts":450115035598946339}' {"state":"normal","resolved_ts":450115035598946339,"checkpoint_ts":450115035598946339} ++ echo '{"state":"normal","resolved_ts":450115035598946339,"checkpoint_ts":450115035598946339}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] + [[ -z last_error ]] ++ echo '{"state":"normal","resolved_ts":450115035598946339,"checkpoint_ts":450115035598946339}' ++ jq -r .last_error.message + error_msg=null + [[ ! null =~ null ]] run task successfully check_changefeed_status 127.0.0.1:8300 1e309ec7-2940-4c98-86d7-40ee911ac24f normal last_warning null + endpoint=127.0.0.1:8300 + changefeed_id=1e309ec7-2940-4c98-86d7-40ee911ac24f + expected_state=normal + field=last_warning + error_pattern=null ++ curl 127.0.0.1:8300/api/v2/changefeeds/1e309ec7-2940-4c98-86d7-40ee911ac24f/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 86 100 86 0 0 482 0 --:--:-- --:--:-- --:--:-- 483 + info='{"state":"normal","resolved_ts":450115035598946339,"checkpoint_ts":450115035598946339}' + echo '{"state":"normal","resolved_ts":450115035598946339,"checkpoint_ts":450115035598946339}' {"state":"normal","resolved_ts":450115035598946339,"checkpoint_ts":450115035598946339} ++ echo '{"state":"normal","resolved_ts":450115035598946339,"checkpoint_ts":450115035598946339}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] + [[ -z last_warning ]] ++ echo '{"state":"normal","resolved_ts":450115035598946339,"checkpoint_ts":450115035598946339}' ++ jq -r .last_warning.message + error_msg=null + [[ ! null =~ null ]] run task successfully table sink_hang.t1 exists table sink_hang.t2 not exists for 1-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 4876 0 --:--:-- --:--:-- --:--:-- 4959 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-30 15:03:16.505","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"2024-05-30 15:03:09.754","now_ts":"2024-05-30 15:03:16.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:03:16.505","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-05-30' '15:03:09.754","now_ts":"2024-05-30' '15:03:16.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:03:16.505","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-05-30' '15:03:09.754","now_ts":"2024-05-30' '15:03:16.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 table multi_tables_ddl_test.finish_mark exists check table exists success table mark.finish_mark_2 not exists for 9-th check, retry later check_redo_resolved_ts 62a1737f-249d-41ad-8fac-3463e7c296aa 450115032963874819 file:///tmp/tidb_cdc_test/consistent_replicate_storage_file/redo /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc_data/redo/62a1737f-249d-41ad-8fac-3463e7c296aa/meta [2024/05/30 15:03:18.626 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0016ca1e0/127.0.0.1:2379] [attempt=0] [error="rpc error: code = NotFound desc = etcdserver: requested lease not found"] global resolved ts 450115032609980425 not forward to 450115032963874819 run task failed 4-th time, retry later + endpoints=http://127.0.0.1:2379 + changefeed_id=test-normal + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c test-normal -s + info='{ "upstream_id": 7374684582254680420, "namespace": "default", "id": "test-normal", "state": "normal", "checkpoint_tso": 450115035104018444, "checkpoint_time": "2024-05-30 15:03:13.628", "error": null }' + echo '{ "upstream_id": 7374684582254680420, "namespace": "default", "id": "test-normal", "state": "normal", "checkpoint_tso": 450115035104018444, "checkpoint_time": "2024-05-30 15:03:13.628", "error": null }' { "upstream_id": 7374684582254680420, "namespace": "default", "id": "test-normal", "state": "normal", "checkpoint_tso": 450115035104018444, "checkpoint_time": "2024-05-30 15:03:13.628", "error": null } ++ echo '{' '"upstream_id":' 7374684582254680420, '"namespace":' '"default",' '"id":' '"test-normal",' '"state":' '"normal",' '"checkpoint_tso":' 450115035104018444, '"checkpoint_time":' '"2024-05-30' '15:03:13.628",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374684582254680420, '"namespace":' '"default",' '"id":' '"test-normal",' '"state":' '"normal",' '"checkpoint_tso":' 450115035104018444, '"checkpoint_time":' '"2024-05-30' '15:03:13.628",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + endpoints=http://127.0.0.1:2379 + changefeed_id=test-error-1 + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c test-error-1 -s + info='{ "upstream_id": 7374684582254680420, "namespace": "default", "id": "test-error-1", "state": "normal", "checkpoint_tso": 450115036270559233, "checkpoint_time": "2024-05-30 15:03:18.078", "error": null }' + echo '{ "upstream_id": 7374684582254680420, "namespace": "default", "id": "test-error-1", "state": "normal", "checkpoint_tso": 450115036270559233, "checkpoint_time": "2024-05-30 15:03:18.078", "error": null }' { "upstream_id": 7374684582254680420, "namespace": "default", "id": "test-error-1", "state": "normal", "checkpoint_tso": 450115036270559233, "checkpoint_time": "2024-05-30 15:03:18.078", "error": null } ++ echo '{' '"upstream_id":' 7374684582254680420, '"namespace":' '"default",' '"id":' '"test-error-1",' '"state":' '"normal",' '"checkpoint_tso":' 450115036270559233, '"checkpoint_time":' '"2024-05-30' '15:03:18.078",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374684582254680420, '"namespace":' '"default",' '"id":' '"test-error-1",' '"state":' '"normal",' '"checkpoint_tso":' 450115036270559233, '"checkpoint_time":' '"2024-05-30' '15:03:18.078",' '"error":' null '}' ++ jq -r .error.message check_ts_forward baafe3a1-81aa-4051-9d66-9bafc75e9067 Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + message=null + [[ ! null =~ null ]] + endpoints=http://127.0.0.1:2379 + changefeed_id=test-error-2 + expected_state=failed + error_msg=ErrSyncRenameTableFailed + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c test-error-2 -s + info='{ "upstream_id": 7374684582254680420, "namespace": "default", "id": "test-error-2", "state": "failed", "checkpoint_tso": 450115034684588040, "checkpoint_time": "2024-05-30 15:03:12.028", "error": { "time": "2024-05-30T15:03:14.112510005+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSyncRenameTableFailed", "message": "[CDC:ErrSyncRenameTableFailed]table'\''s old name is not in filter rule, and its new name in filter rule table id '\''128'\'', ddl query: [rename table t11 to t9], it'\''s an unexpected behavior, if you want to replicate this table, please add its old name to filter rule." } }' + echo '{ "upstream_id": 7374684582254680420, "namespace": "default", "id": "test-error-2", "state": "failed", "checkpoint_tso": 450115034684588040, "checkpoint_time": "2024-05-30 15:03:12.028", "error": { "time": "2024-05-30T15:03:14.112510005+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSyncRenameTableFailed", "message": "[CDC:ErrSyncRenameTableFailed]table'\''s old name is not in filter rule, and its new name in filter rule table id '\''128'\'', ddl query: [rename table t11 to t9], it'\''s an unexpected behavior, if you want to replicate this table, please add its old name to filter rule." } }' { "upstream_id": 7374684582254680420, "namespace": "default", "id": "test-error-2", "state": "failed", "checkpoint_tso": 450115034684588040, "checkpoint_time": "2024-05-30 15:03:12.028", "error": { "time": "2024-05-30T15:03:14.112510005+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSyncRenameTableFailed", "message": "[CDC:ErrSyncRenameTableFailed]table's old name is not in filter rule, and its new name in filter rule table id '128', ddl query: [rename table t11 to t9], it's an unexpected behavior, if you want to replicate this table, please add its old name to filter rule." } } ++ echo '{' '"upstream_id":' 7374684582254680420, '"namespace":' '"default",' '"id":' '"test-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 450115034684588040, '"checkpoint_time":' '"2024-05-30' '15:03:12.028",' '"error":' '{' '"time":' '"2024-05-30T15:03:14.112510005+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSyncRenameTableFailed",' '"message":' '"[CDC:ErrSyncRenameTableFailed]table'\''s' old name is not in filter rule, and its new name in filter rule table id ''\''128'\'',' ddl query: '[rename' table t11 to 't9],' 'it'\''s' an unexpected behavior, if you want to replicate this table, please add its old name to filter 'rule."' '}' '}' ++ jq -r .state + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ jq -r .error.message ++ echo '{' '"upstream_id":' 7374684582254680420, '"namespace":' '"default",' '"id":' '"test-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 450115034684588040, '"checkpoint_time":' '"2024-05-30' '15:03:12.028",' '"error":' '{' '"time":' '"2024-05-30T15:03:14.112510005+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSyncRenameTableFailed",' '"message":' '"[CDC:ErrSyncRenameTableFailed]table'\''s' old name is not in filter rule, and its new name in filter rule table id ''\''128'\'',' ddl query: '[rename' table t11 to 't9],' 'it'\''s' an unexpected behavior, if you want to replicate this table, please add its old name to filter 'rule."' '}' '}' + message='[CDC:ErrSyncRenameTableFailed]table'\''s old name is not in filter rule, and its new name in filter rule table id '\''128'\'', ddl query: [rename table t11 to t9], it'\''s an unexpected behavior, if you want to replicate this table, please add its old name to filter rule.' + [[ ! [CDC:ErrSyncRenameTableFailed]table's old name is not in filter rule, and its new name in filter rule table id '128', ddl query: [rename table t11 to t9], it's an unexpected behavior, if you want to replicate this table, please add its old name to filter rule. =~ ErrSyncRenameTableFailed ]] check diff successfully wait process cdc.test exit for 1-th time... table mark.finish_mark_2 not exists for 10-th check, retry later wait process cdc.test exit for 2-th time... changefeed is working normally rts: 450115036388524000->450115036650668000 checkpoint: 450115036388524000->450115036650668000 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql true [2024/05/30 15:03:14.912 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="CREATE TABLE `ddl_reentrant`.`t4` (`a` INT PRIMARY KEY,`b` INT)"] [duration=86.498551ms] [namespace=default] [changefeed=baafe3a1-81aa-4051-9d66-9bafc75e9067] run task successfully table sink_hang.t2 exists check diff failed 1-th time, retry later wait process cdc.test exit for 3-th time... resume changefeed baafe3a1-81aa-4051-9d66-9bafc75e9067 from 450115035300626433 check_ts_forward baafe3a1-81aa-4051-9d66-9bafc75e9067 wait process 7785 exit for 1-th time... wait process 7785 exit for 2-th time... wait process 7785 exit for 3-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (7785) - No such process wait process 7785 exit for 4-th time... process 7785 already exit check_capture_count http://127.0.0.1:2379 0 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Thu May 30 15:03:21 CST 2024] <<<<<< START cdc server in kill_owner_with_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql/MySQLSinkExecDDLDelay=return(true);github.com/pingcap/tiflow/cdc/capture/ownerFlushIntervalInject=return(10)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kill_owner_with_ddl.79477949.out server --log-file /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:03:21 CST 2024] <<<<<< run test case multi_tables_ddl success! >>>>>> Exiting on signal: INTERRUPT table mark.finish_mark_2 not exists for 11-th check, retry later check diff failed 2-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) run task failed 1-th time, retry later [2024/05/30 15:03:18.626 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0019261e0/127.0.0.1:2379] [attempt=0] [error="rpc error: code = NotFound desc = etcdserver: requested lease not found"] [2024/05/30 15:03:18.627 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0015881e0/127.0.0.1:2479] [attempt=0] [error="rpc error: code = NotFound desc = etcdserver: requested lease not found"] table mark.finish_mark_2 not exists for 12-th check, retry later [2024/05/30 15:03:24.679 +08:00] [INFO] [main.go:86] ["running ddl test: 0 createDropSchemaDDL"] [2024/05/30 15:03:24.933 +08:00] [INFO] [main.go:220] ["0 insert success: 100"] [2024/05/30 15:03:24.938 +08:00] [INFO] [main.go:220] ["1 insert success: 100"] check diff failed 3-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:03:24 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/743c206c-db17-4f29-b103-9e56a2826b2d {UpstreamID:7374684617514448989 Namespace:default ID:743c206c-db17-4f29-b103-9e56a2826b2d SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-30 15:03:04.811090645 +0800 CST StartTs:450115032747606021 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003792f30 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115032786927619} {CheckpointTs:450115035014889498 MinTableBarrierTs:450115035014889498 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115035014889498, checkpointTs: 450115035014889498, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dc8ce19f-4edb-4c5a-8b7c-1b8b47ce1e8a {"id":"dc8ce19f-4edb-4c5a-8b7c-1b8b47ce1e8a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ee1f8bb dc8ce19f-4edb-4c5a-8b7c-1b8b47ce1e8a /tidb/cdc/default/default/changefeed/info/743c206c-db17-4f29-b103-9e56a2826b2d {"upstream-id":7374684617514448989,"namespace":"default","changefeed-id":"743c206c-db17-4f29-b103-9e56a2826b2d","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T15:03:04.811090645+08:00","start-ts":450115032747606021,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115032786927619} /tidb/cdc/default/default/changefeed/status/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":450115035014889498,"min-table-barrier-ts":450115035014889498,"admin-job-type":0} /tidb/cdc/default/default/task/position/dc8ce19f-4edb-4c5a-8b7c-1b8b47ce1e8a/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684617514448989 {"id":7374684617514448989,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/743c206c-db17-4f29-b103-9e56a2826b2d {UpstreamID:7374684617514448989 Namespace:default ID:743c206c-db17-4f29-b103-9e56a2826b2d SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-30 15:03:04.811090645 +0800 CST StartTs:450115032747606021 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003792f30 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115032786927619} {CheckpointTs:450115035014889498 MinTableBarrierTs:450115035014889498 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115035014889498, checkpointTs: 450115035014889498, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dc8ce19f-4edb-4c5a-8b7c-1b8b47ce1e8a {"id":"dc8ce19f-4edb-4c5a-8b7c-1b8b47ce1e8a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ee1f8bb dc8ce19f-4edb-4c5a-8b7c-1b8b47ce1e8a /tidb/cdc/default/default/changefeed/info/743c206c-db17-4f29-b103-9e56a2826b2d {"upstream-id":7374684617514448989,"namespace":"default","changefeed-id":"743c206c-db17-4f29-b103-9e56a2826b2d","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T15:03:04.811090645+08:00","start-ts":450115032747606021,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115032786927619} /tidb/cdc/default/default/changefeed/status/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":450115035014889498,"min-table-barrier-ts":450115035014889498,"admin-job-type":0} /tidb/cdc/default/default/task/position/dc8ce19f-4edb-4c5a-8b7c-1b8b47ce1e8a/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684617514448989 {"id":7374684617514448989,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/743c206c-db17-4f29-b103-9e56a2826b2d {UpstreamID:7374684617514448989 Namespace:default ID:743c206c-db17-4f29-b103-9e56a2826b2d SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-30 15:03:04.811090645 +0800 CST StartTs:450115032747606021 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003792f30 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115032786927619} {CheckpointTs:450115035014889498 MinTableBarrierTs:450115035014889498 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115035014889498, checkpointTs: 450115035014889498, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dc8ce19f-4edb-4c5a-8b7c-1b8b47ce1e8a {"id":"dc8ce19f-4edb-4c5a-8b7c-1b8b47ce1e8a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ee1f8bb dc8ce19f-4edb-4c5a-8b7c-1b8b47ce1e8a /tidb/cdc/default/default/changefeed/info/743c206c-db17-4f29-b103-9e56a2826b2d {"upstream-id":7374684617514448989,"namespace":"default","changefeed-id":"743c206c-db17-4f29-b103-9e56a2826b2d","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T15:03:04.811090645+08:00","start-ts":450115032747606021,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115032786927619} /tidb/cdc/default/default/changefeed/status/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":450115035014889498,"min-table-barrier-ts":450115035014889498,"admin-job-type":0} /tidb/cdc/default/default/task/position/dc8ce19f-4edb-4c5a-8b7c-1b8b47ce1e8a/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684617514448989 + grep -q 'etcd info' {"id":7374684617514448989,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x check_capture_count http://127.0.0.1:2379 1 run task successfully [2024/05/30 15:03:25.131 +08:00] [INFO] [main.go:220] ["0 insert success: 200"] [2024/05/30 15:03:25.134 +08:00] [INFO] [main.go:234] ["0 delete success: 100"] [2024/05/30 15:03:25.166 +08:00] [INFO] [main.go:234] ["1 delete success: 100"] [2024/05/30 15:03:25.167 +08:00] [INFO] [main.go:220] ["1 insert success: 200"] check_ts_forward baafe3a1-81aa-4051-9d66-9bafc75e9067 table mark.finish_mark_2 not exists for 13-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213e03c0000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t2zx2, pid:20452, start at 2024-05-30 15:03:24.671633372 +0800 CST m=+5.323119360 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:05:24.679 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:03:24.656 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:53:24.656 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213e03c0000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t2zx2, pid:20452, start at 2024-05-30 15:03:24.671633372 +0800 CST m=+5.323119360 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:05:24.679 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:03:24.656 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:53:24.656 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213e04940003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t2zx2, pid:20532, start at 2024-05-30 15:03:24.711757548 +0800 CST m=+5.278370072 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:05:24.720 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:03:24.709 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:53:24.709 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/move_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/move_table/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/move_table/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/move_table/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/move_table/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2024/05/30 15:03:26.421 +08:00] [INFO] [main.go:220] ["1 insert success: 300"] [2024/05/30 15:03:26.472 +08:00] [INFO] [main.go:220] ["0 insert success: 300"] [2024/05/30 15:03:26.697 +08:00] [INFO] [main.go:234] ["1 delete success: 200"] [2024/05/30 15:03:26.700 +08:00] [INFO] [main.go:220] ["1 insert success: 400"] [2024/05/30 15:03:26.730 +08:00] [INFO] [main.go:220] ["0 insert success: 400"] [2024/05/30 15:03:26.732 +08:00] [INFO] [main.go:234] ["0 delete success: 200"] check_redo_resolved_ts 62a1737f-249d-41ad-8fac-3463e7c296aa 450115032963874819 file:///tmp/tidb_cdc_test/consistent_replicate_storage_file/redo /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc_data/redo/62a1737f-249d-41ad-8fac-3463e7c296aa/meta run task successfully [2024/05/30 15:03:26.943 +08:00] [INFO] [main.go:220] ["1 insert success: 500"] wait process cdc.test exit for 1-th time... check diff failed 4-th time, retry later changefeed is working normally rts: 450115037961388000->450115038485676000 checkpoint: 450115037961388000->450115038485676000 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql false [2024/05/30 15:03:21.684 +08:00] [INFO] [mysql_ddl_sink.go:119] ["Execute DDL failed, but error can be ignored"] [startTs=450115035287519257] [ddl="CREATE TABLE `ddl_reentrant`.`t4` (`a` INT PRIMARY KEY,`b` INT)"] [namespace=default] [changefeed=baafe3a1-81aa-4051-9d66-9bafc75e9067] [error="Error 1050 (42S01): Table 'ddl_reentrant.t4' already exists"] run task successfully ------------------------------------------ test ddl alter table ddl_reentrant.t3 exchange partition p0 with table ddl_reentrant.t4, is_reentrant: true restored_sql: ALTER TABLE `ddl_reentrant`.`t3` EXCHANGE PARTITION `p0` WITH TABLE `ddl_reentrant`.`t4` [2024/05/30 15:03:28.016 +08:00] [INFO] [main.go:220] ["0 insert success: 500"] wait process cdc.test exit for 2-th time... check_ts_forward baafe3a1-81aa-4051-9d66-9bafc75e9067 [2024/05/30 15:03:28.174 +08:00] [INFO] [main.go:234] ["1 delete success: 300"] [2024/05/30 15:03:28.179 +08:00] [INFO] [main.go:220] ["1 insert success: 600"] [2024/05/30 15:03:28.233 +08:00] [INFO] [main.go:220] ["0 insert success: 600"] [2024/05/30 15:03:28.242 +08:00] [INFO] [main.go:234] ["0 delete success: 300"] wait process cdc.test exit for 3-th time... [2024/05/30 15:03:28.394 +08:00] [INFO] [main.go:220] ["1 insert success: 700"] [2024/05/30 15:03:28.475 +08:00] [INFO] [main.go:220] ["0 insert success: 700"] table mark.finish_mark_2 exists table mark.finish_mark_3 not exists for 1-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [2024/05/30 15:03:28.865 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2024/05/30 15:03:28.866 +08:00] [INFO] [apply.go:78] ["set memory limit"] [memoryLimit=10737418240] [2024/05/30 15:03:28.866 +08:00] [INFO] [apply.go:94] ["Start http pprof server"] [addr=:6060] [2024/05/30 15:03:28.866 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2024/05/30 15:03:28.866 +08:00] [INFO] [redo.go:167] ["apply redo log starts"] [checkpointTs=450115025623580698] [resolvedTs=450115038142267397] [2024/05/30 15:03:28.870 +08:00] [INFO] [tz.go:36] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/05/30 15:03:28.870 +08:00] [WARN] [config.go:394] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/05/30 15:03:28.871 +08:00] [ERROR] [config.go:209] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.cli.21840.out cli tso query --pd=http://127.0.0.1:2379 [2024/05/30 15:03:28.871 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=4.933706ms] [2024/05/30 15:03:28.878 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22IGNORE_SPACE%2CONLY_FULL_GROUP_BY%2CNO_ENGINE_SUBSTITUTION%2CNO_AUTO_VALUE_ON_ZERO%2CALLOW_INVALID_DATES%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/05/30 15:03:28.882 +08:00] [INFO] [mysql.go:198] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2024/05/30 15:03:28.883 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:03:28.883 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:03:28.883 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:03:28.883 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:03:28.883 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:03:28.883 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:03:28.883 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:03:28.883 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:03:28.883 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:03:28.883 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:03:28.883 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:03:28.883 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:03:28.883 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:03:28.883 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:03:28.883 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:03:28.883 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:03:28.883 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2024/05/30 15:03:28.883 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2024/05/30 15:03:28.883 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2024/05/30 15:03:28.883 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2024/05/30 15:03:28.883 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2024/05/30 15:03:28.883 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2024/05/30 15:03:28.883 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2024/05/30 15:03:28.883 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2024/05/30 15:03:28.883 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2024/05/30 15:03:28.883 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2024/05/30 15:03:28.883 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2024/05/30 15:03:28.883 +08:00] [INFO] [tz.go:36] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/05/30 15:03:28.883 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2024/05/30 15:03:28.883 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2024/05/30 15:03:28.883 +08:00] [WARN] [config.go:394] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/05/30 15:03:28.883 +08:00] [ERROR] [config.go:209] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2024/05/30 15:03:28.883 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2024/05/30 15:03:28.883 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2024/05/30 15:03:28.883 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2024/05/30 15:03:28.889 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22NO_ENGINE_SUBSTITUTION%2CIGNORE_SPACE%2CONLY_FULL_GROUP_BY%2CALLOW_INVALID_DATES%2CNO_AUTO_VALUE_ON_ZERO%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/05/30 15:03:28.893 +08:00] [INFO] [mysql_ddl_sink.go:99] ["MySQL DDL sink is created"] [namespace=] [changefeed=] check diff failed 5-th time, retry later [2024/05/30 15:03:29.403 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=536.357389ms] [2024/05/30 15:03:29.404 +08:00] [ERROR] [redo.go:273] ["ignore unsupported DDL"] [ddl="{\"StartTs\":450115020551094279,\"CommitTs\":450115025623580698,\"Query\":\"ALTER TABLE `consistent_replicate_storage_file`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `consistent_replicate_storage_file`.`t2`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_storage_file\",\"tbl-name\":\"t1\",\"tbl-id\":108,\"is-partition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":42,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [2024/05/30 15:03:29.405 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=283] [newQuota=72448] [2024/05/30 15:03:29.420 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=291] [newQuota=74496] [2024/05/30 15:03:29.437 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=291] [newQuota=74496] [2024/05/30 15:03:29.453 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=291] [newQuota=74496] [2024/05/30 15:03:29.473 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=291] [newQuota=74496] [2024/05/30 15:03:29.523 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=291] [newQuota=74496] [2024/05/30 15:03:29.545 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=291] [newQuota=74496] [2024/05/30 15:03:29.560 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=291] [newQuota=74496] [2024/05/30 15:03:29.622 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=291] [newQuota=74496] [2024/05/30 15:03:29.643 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=291] [newQuota=74496] [2024/05/30 15:03:29.663 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=291] [newQuota=74496] [2024/05/30 15:03:29.650 +08:00] [INFO] [main.go:234] ["1 delete success: 400"] [2024/05/30 15:03:29.651 +08:00] [INFO] [main.go:220] ["1 insert success: 800"] [2024/05/30 15:03:29.730 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=291] [newQuota=74496] [2024/05/30 15:03:29.752 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=291] [newQuota=74496] [2024/05/30 15:03:29.778 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=291] [newQuota=74496] [2024/05/30 15:03:29.837 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=291] [newQuota=74496] [2024/05/30 15:03:29.857 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=291] [newQuota=74496] [2024/05/30 15:03:29.923 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=291] [newQuota=74496] [2024/05/30 15:03:29.948 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=291] [newQuota=74496] [2024/05/30 15:03:30.026 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=291] [newQuota=74496] [2024/05/30 15:03:30.038 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=291] [newQuota=74496] + set +x + tso='450115039017304070 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115039017304070 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ***************** properties ***************** "mysql.port"="4000" "mysql.db"="move_table" "workload"="core" "operationcount"="0" "scanproportion"="0" "mysql.host"="127.0.0.1" "readallfields"="true" "threadcount"="10" "mysql.user"="root" "requestdistribution"="uniform" "insertproportion"="0" "updateproportion"="0" "recordcount"="10000" "dotransactions"="false" "readproportion"="0" ********************************************** table mark.finish_mark_3 not exists for 2-th check, retry later changefeed is working normally rts: 450115038747820000->450115039009964000 checkpoint: 450115038747820000->450115039272108000 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql true [2024/05/30 15:03:28.029 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="ALTER TABLE `ddl_reentrant`.`t3` EXCHANGE PARTITION `p0` WITH TABLE `ddl_reentrant`.`t4`"] [duration=149.497583ms] [namespace=default] [changefeed=baafe3a1-81aa-4051-9d66-9bafc75e9067] run task successfully resume changefeed baafe3a1-81aa-4051-9d66-9bafc75e9067 from 450115038734974987 check_ts_forward baafe3a1-81aa-4051-9d66-9bafc75e9067 [2024/05/30 15:03:30.848 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:118,start_key:7480000000000000ff765f720000000000fa,end_key:7480000000000000ff765f730000000000fa}] [checkpointTs=450115038142267397] [2024/05/30 15:03:30.848 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:118,start_key:7480000000000000ff765f720000000000fa,end_key:7480000000000000ff765f730000000000fa}] [checkpointTs=450115038142267397] [2024/05/30 15:03:30.949 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}] [checkpointTs=450115038142267397] [2024/05/30 15:03:30.949 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}] [checkpointTs=450115038142267397] [2024/05/30 15:03:31.049 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}] [checkpointTs=450115038142267397] [2024/05/30 15:03:31.050 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}] [checkpointTs=450115038142267397] check diff failed 6-th time, retry later [2024/05/30 15:03:31.150 +08:00] [INFO] [redo.go:223] ["apply redo log finishes"] [appliedLogCount=5008] [appliedDDLCount=0] [currentCheckpoint=450115038142267397] [2024/05/30 15:03:31.150 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2024/05/30 15:03:31.151 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2024/05/30 15:03:31.151 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2024/05/30 15:03:31.151 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2024/05/30 15:03:31.151 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2024/05/30 15:03:31.151 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2024/05/30 15:03:31.151 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] [2024/05/30 15:03:31.151 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2024/05/30 15:03:31.151 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] [2024/05/30 15:03:31.151 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] [2024/05/30 15:03:31.151 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2024/05/30 15:03:31.151 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2024/05/30 15:03:31.151 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2024/05/30 15:03:31.151 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2024/05/30 15:03:31.151 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2024/05/30 15:03:31.151 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] Apply redo log successfully \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } table mark.finish_mark_3 not exists for 3-th check, retry later [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } run task failed 1-th time, retry later check diff successfully [Thu May 30 15:03:32 CST 2024] <<<<<< run test case consistent_replicate_storage_file success! >>>>>> count(*) 5000 Run finished, takes 3.189157996s INSERT - Takes(s): 3.2, Count: 10000, OPS: 3141.4, Avg(us): 3136, Min(us): 790, Max(us): 85028, 95th(us): 5000, 99th(us): 60000 [Thu May 30 15:03:33 CST 2024] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.2191521917.out server --log-file /tmp/tidb_cdc_test/move_table/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data1 --cluster-id default --addr 127.0.0.1:8300 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 7-th time, retry later table big_txn.finish_mark_1 exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:03:27 CST 2024] <<<<<< run test case big_txn success! >>>>>> table mark.finish_mark_3 not exists for 4-th check, retry later check_ts_forward baafe3a1-81aa-4051-9d66-9bafc75e9067 wait process 7952 exit for 1-th time... wait process 7952 exit for 2-th time... wait process 7952 exit for 3-th time... /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (7952) - No such process wait process 7952 exit for 4-th time... process 7952 already exit check_capture_count http://127.0.0.1:2379 0 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Thu May 30 15:03:35 CST 2024] <<<<<< START cdc server in kill_owner_with_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql/MySQLSinkExecDDLDelay=return(true);github.com/pingcap/tiflow/cdc/capture/ownerFlushIntervalInject=return(10)' + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kill_owner_with_ddl.80858087.out server --log-file /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 8-th time, retry later [2024/05/30 15:03:35.229 +08:00] [INFO] [main.go:86] ["running ddl test: 1 truncateDDL"] [2024/05/30 15:03:35.462 +08:00] [INFO] [main.go:220] ["0 insert success: 100"] [2024/05/30 15:03:35.467 +08:00] [INFO] [main.go:220] ["1 insert success: 100"] [2024/05/30 15:03:35.648 +08:00] [INFO] [main.go:220] ["0 insert success: 200"] [2024/05/30 15:03:35.650 +08:00] [INFO] [main.go:234] ["0 delete success: 100"] [2024/05/30 15:03:35.660 +08:00] [INFO] [main.go:220] ["1 insert success: 200"] [2024/05/30 15:03:35.662 +08:00] [INFO] [main.go:234] ["1 delete success: 100"] [2024/05/30 15:03:35.865 +08:00] [INFO] [main.go:220] ["0 insert success: 300"] [2024/05/30 15:03:35.936 +08:00] [INFO] [main.go:220] ["1 insert success: 300"] [2024/05/30 15:03:36.064 +08:00] [INFO] [main.go:220] ["0 insert success: 400"] [2024/05/30 15:03:36.066 +08:00] [INFO] [main.go:234] ["0 delete success: 200"] [2024/05/30 15:03:36.135 +08:00] [INFO] [main.go:220] ["1 insert success: 400"] [2024/05/30 15:03:36.138 +08:00] [INFO] [main.go:234] ["1 delete success: 200"] [2024/05/30 15:03:36.283 +08:00] [INFO] [main.go:220] ["0 insert success: 500"] [2024/05/30 15:03:36.342 +08:00] [INFO] [main.go:220] ["1 insert success: 500"] [2024/05/30 15:03:36.486 +08:00] [INFO] [main.go:220] ["0 insert success: 600"] [2024/05/30 15:03:36.488 +08:00] [INFO] [main.go:234] ["0 delete success: 300"] [2024/05/30 15:03:36.549 +08:00] [INFO] [main.go:220] ["1 insert success: 600"] [2024/05/30 15:03:36.556 +08:00] [INFO] [main.go:234] ["1 delete success: 300"] [2024/05/30 15:03:36.680 +08:00] [INFO] [main.go:220] ["0 insert success: 700"] changefeed is working normally rts: 450115039272108000->450115040844972000 checkpoint: 450115040582828000->450115040844972000 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql true [2024/05/30 15:03:32.833 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="ALTER TABLE `ddl_reentrant`.`t3` EXCHANGE PARTITION `p0` WITH TABLE `ddl_reentrant`.`t4`"] [duration=158.317229ms] [namespace=default] [changefeed=baafe3a1-81aa-4051-9d66-9bafc75e9067] run task successfully [2024/05/30 15:03:36.743 +08:00] [INFO] [main.go:220] ["1 insert success: 700"] [2024/05/30 15:03:36.866 +08:00] [INFO] [main.go:220] ["0 insert success: 800"] [2024/05/30 15:03:36.868 +08:00] [INFO] [main.go:234] ["0 delete success: 400"] [2024/05/30 15:03:36.933 +08:00] [INFO] [main.go:220] ["1 insert success: 800"] [2024/05/30 15:03:36.938 +08:00] [INFO] [main.go:234] ["1 delete success: 400"] table mark.finish_mark_3 not exists for 5-th check, retry later [2024/05/30 15:03:37.065 +08:00] [INFO] [main.go:220] ["0 insert success: 900"] [2024/05/30 15:03:37.136 +08:00] [INFO] [main.go:220] ["1 insert success: 900"] [2024/05/30 15:03:37.263 +08:00] [INFO] [main.go:220] ["0 insert success: 1000"] [2024/05/30 15:03:37.265 +08:00] [INFO] [main.go:234] ["0 delete success: 500"] [2024/05/30 15:03:37.325 +08:00] [INFO] [main.go:220] ["1 insert success: 1000"] [2024/05/30 15:03:37.331 +08:00] [INFO] [main.go:234] ["1 delete success: 500"] wait process cdc.test exit for 1-th time... [2024/05/30 15:03:37.459 +08:00] [INFO] [main.go:220] ["0 insert success: 1100"] [2024/05/30 15:03:37.536 +08:00] [INFO] [main.go:220] ["1 insert success: 1100"] [2024/05/30 15:03:37.679 +08:00] [INFO] [main.go:220] ["0 insert success: 1200"] [2024/05/30 15:03:37.681 +08:00] [INFO] [main.go:234] ["0 delete success: 600"] wait process cdc.test exit for 2-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:03:37 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b276fcaf-764a-411e-ae34-b73a9c1ed300 {"id":"b276fcaf-764a-411e-ae34-b73a9c1ed300","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f5847e2 b276fcaf-764a-411e-ae34-b73a9c1ed300 /tidb/cdc/default/default/upstream/7374684746182983475 {"id":7374684746182983475,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b276fcaf-764a-411e-ae34-b73a9c1ed300 {"id":"b276fcaf-764a-411e-ae34-b73a9c1ed300","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f5847e2 b276fcaf-764a-411e-ae34-b73a9c1ed300 /tidb/cdc/default/default/upstream/7374684746182983475 {"id":7374684746182983475,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b276fcaf-764a-411e-ae34-b73a9c1ed300 {"id":"b276fcaf-764a-411e-ae34-b73a9c1ed300","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f5847e2 b276fcaf-764a-411e-ae34-b73a9c1ed300 /tidb/cdc/default/default/upstream/7374684746182983475 {"id":7374684746182983475,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.cli.21958.out cli changefeed create --start-ts=450115039017304070 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' [2024/05/30 15:03:37.850 +08:00] [INFO] [main.go:220] ["1 insert success: 1200"] [2024/05/30 15:03:37.856 +08:00] [INFO] [main.go:234] ["1 delete success: 600"] [2024/05/30 15:03:37.881 +08:00] [INFO] [main.go:220] ["0 insert success: 1300"] Create changefeed successfully! ID: b51e63e2-45ad-4f8b-b762-ba2a49072db9 Info: {"upstream_id":7374684746182983475,"namespace":"default","id":"b51e63e2-45ad-4f8b-b762-ba2a49072db9","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T15:03:37.962726757+08:00","start_ts":450115039017304070,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115039017304070,"checkpoint_ts":450115039017304070,"checkpoint_time":"2024-05-30 15:03:28.556"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... check diff successfully wait process cdc.test exit for 1-th time... [2024/05/30 15:03:38.038 +08:00] [INFO] [main.go:220] ["1 insert success: 1300"] [2024/05/30 15:03:38.066 +08:00] [INFO] [main.go:220] ["0 insert success: 1400"] [2024/05/30 15:03:38.068 +08:00] [INFO] [main.go:234] ["0 delete success: 700"] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 15:03:38 CST 2024] <<<<<< run test case ddl_reentrant success! >>>>>> [2024/05/30 15:03:38.231 +08:00] [INFO] [main.go:220] ["1 insert success: 1400"] [2024/05/30 15:03:38.236 +08:00] [INFO] [main.go:234] ["1 delete success: 700"] [2024/05/30 15:03:38.267 +08:00] [INFO] [main.go:220] ["0 insert success: 1500"] [2024/05/30 15:03:38.422 +08:00] [INFO] [main.go:220] ["1 insert success: 1500"] [2024/05/30 15:03:38.447 +08:00] [INFO] [main.go:220] ["0 insert success: 1600"] [2024/05/30 15:03:38.452 +08:00] [INFO] [main.go:234] ["0 delete success: 800"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:03:38 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/743c206c-db17-4f29-b103-9e56a2826b2d processor is not initialized *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/94c144fa-5dcc-40b5-a344-c540085f6062 {"id":"94c144fa-5dcc-40b5-a344-c540085f6062","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ee1f91e 94c144fa-5dcc-40b5-a344-c540085f6062 /tidb/cdc/default/default/changefeed/info/743c206c-db17-4f29-b103-9e56a2826b2d {"upstream-id":7374684617514448989,"namespace":"default","changefeed-id":"743c206c-db17-4f29-b103-9e56a2826b2d","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T15:03:04.811090645+08:00","start-ts":450115032747606021,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115032786927619} /tidb/cdc/default/default/changefeed/status/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":450115035014889498,"min-table-barrier-ts":450115035014889498,"admin-job-type":0} /tidb/cdc/default/default/task/position/94c144fa-5dcc-40b5-a344-c540085f6062/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684617514448989 {"id":7374684617514448989,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/743c206c-db17-4f29-b103-9e56a2826b2d processor is not initialized *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/94c144fa-5dcc-40b5-a344-c540085f6062 {"id":"94c144fa-5dcc-40b5-a344-c540085f6062","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ee1f91e 94c144fa-5dcc-40b5-a344-c540085f6062 /tidb/cdc/default/default/changefeed/info/743c206c-db17-4f29-b103-9e56a2826b2d {"upstream-id":7374684617514448989,"namespace":"default","changefeed-id":"743c206c-db17-4f29-b103-9e56a2826b2d","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T15:03:04.811090645+08:00","start-ts":450115032747606021,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115032786927619} /tidb/cdc/default/default/changefeed/status/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":450115035014889498,"min-table-barrier-ts":450115035014889498,"admin-job-type":0} /tidb/cdc/default/default/task/position/94c144fa-5dcc-40b5-a344-c540085f6062/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684617514448989 + grep -q 'failed to get info:' {"id":7374684617514448989,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/743c206c-db17-4f29-b103-9e56a2826b2d processor is not initialized *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/94c144fa-5dcc-40b5-a344-c540085f6062 {"id":"94c144fa-5dcc-40b5-a344-c540085f6062","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ee1f91e 94c144fa-5dcc-40b5-a344-c540085f6062 /tidb/cdc/default/default/changefeed/info/743c206c-db17-4f29-b103-9e56a2826b2d {"upstream-id":7374684617514448989,"namespace":"default","changefeed-id":"743c206c-db17-4f29-b103-9e56a2826b2d","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T15:03:04.811090645+08:00","start-ts":450115032747606021,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115032786927619} /tidb/cdc/default/default/changefeed/status/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":450115035014889498,"min-table-barrier-ts":450115035014889498,"admin-job-type":0} /tidb/cdc/default/default/task/position/94c144fa-5dcc-40b5-a344-c540085f6062/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684617514448989 {"id":7374684617514448989,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_capture_count http://127.0.0.1:2379 1 run task successfully [2024/05/30 15:03:38.599 +08:00] [INFO] [main.go:220] ["1 insert success: 1600"] [2024/05/30 15:03:38.604 +08:00] [INFO] [main.go:234] ["1 delete success: 800"] [2024/05/30 15:03:38.631 +08:00] [INFO] [main.go:220] ["0 insert success: 1700"] wait process cdc.test exit for 2-th time... table mark.finish_mark_3 not exists for 6-th check, retry later [2024/05/30 15:03:38.801 +08:00] [INFO] [main.go:220] ["1 insert success: 1700"] [2024/05/30 15:03:38.856 +08:00] [INFO] [main.go:220] ["0 insert success: 1800"] [2024/05/30 15:03:38.865 +08:00] [INFO] [main.go:234] ["0 delete success: 900"] [2024/05/30 15:03:39.008 +08:00] [INFO] [main.go:220] ["1 insert success: 1800"] [2024/05/30 15:03:39.013 +08:00] [INFO] [main.go:234] ["1 delete success: 900"] [2024/05/30 15:03:39.054 +08:00] [INFO] [main.go:220] ["0 insert success: 1900"] [2024/05/30 15:03:39.180 +08:00] [INFO] [main.go:220] ["1 insert success: 1900"] wait process cdc.test exit for 3-th time... [2024/05/30 15:03:39.236 +08:00] [INFO] [main.go:220] ["0 insert success: 2000"] [2024/05/30 15:03:39.245 +08:00] [INFO] [main.go:234] ["0 delete success: 1000"] [2024/05/30 15:03:39.350 +08:00] [INFO] [main.go:220] ["1 insert success: 2000"] [2024/05/30 15:03:39.356 +08:00] [INFO] [main.go:234] ["1 delete success: 1000"] [2024/05/30 15:03:39.420 +08:00] [INFO] [main.go:220] ["0 insert success: 2100"] + set +x [Thu May 30 15:03:39 CST 2024] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.2200322005.out server --log-file /tmp/tidb_cdc_test/move_table/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data2 --cluster-id default --addr 127.0.0.1:8301 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/changefeed_finish/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2024/05/30 15:03:39.549 +08:00] [INFO] [main.go:220] ["1 insert success: 2100"] [2024/05/30 15:03:39.629 +08:00] [INFO] [main.go:220] ["0 insert success: 2200"] [2024/05/30 15:03:39.645 +08:00] [INFO] [main.go:234] ["0 delete success: 1100"] cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:03:39 CST 2024] <<<<<< run test case sink_hang success! >>>>>> [2024/05/30 15:03:39.755 +08:00] [INFO] [main.go:220] ["1 insert success: 2200"] [2024/05/30 15:03:39.763 +08:00] [INFO] [main.go:234] ["1 delete success: 1100"] [2024/05/30 15:03:39.831 +08:00] [INFO] [main.go:220] ["0 insert success: 2300"] [2024/05/30 15:03:39.959 +08:00] [INFO] [main.go:220] ["1 insert success: 2300"] [2024/05/30 15:03:40.040 +08:00] [INFO] [main.go:220] ["0 insert success: 2400"] [2024/05/30 15:03:40.066 +08:00] [INFO] [main.go:234] ["0 delete success: 1200"] [2024/05/30 15:03:40.185 +08:00] [INFO] [main.go:220] ["1 insert success: 2400"] [2024/05/30 15:03:40.200 +08:00] [INFO] [main.go:234] ["1 delete success: 1200"] table mark.finish_mark_3 not exists for 7-th check, retry later table mark.finish_mark_3 not exists for 8-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:03:42 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {UpstreamID:7374684746182983475 Namespace:default ID:b51e63e2-45ad-4f8b-b762-ba2a49072db9 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-30 15:03:37.962726757 +0800 CST StartTs:450115039017304070 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003fde1b0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115041481457666} {CheckpointTs:450115039502270642 MinTableBarrierTs:450115042398961667 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b276fcaf-764a-411e-ae34-b73a9c1ed300 {"id":"b276fcaf-764a-411e-ae34-b73a9c1ed300","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/f356eb82-478a-4ffb-81bf-d0870c09ca41 {"id":"f356eb82-478a-4ffb-81bf-d0870c09ca41","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f5847e2 b276fcaf-764a-411e-ae34-b73a9c1ed300 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f584834 f356eb82-478a-4ffb-81bf-d0870c09ca41 /tidb/cdc/default/default/changefeed/info/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"upstream-id":7374684746182983475,"namespace":"default","changefeed-id":"b51e63e2-45ad-4f8b-b762-ba2a49072db9","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T15:03:37.962726757+08:00","start-ts":450115039017304070,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115041481457666} /tidb/cdc/default/default/changefeed/status/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"checkpoint-ts":450115039502270642,"min-table-barrier-ts":450115042398961667,"admin-job-type":0} /tidb/cdc/default/default/task/position/b276fcaf-764a-411e-ae34-b73a9c1ed300/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f356eb82-478a-4ffb-81bf-d0870c09ca41/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684746182983475 {"id":7374684746182983475,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {UpstreamID:7374684746182983475 Namespace:default ID:b51e63e2-45ad-4f8b-b762-ba2a49072db9 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-30 15:03:37.962726757 +0800 CST StartTs:450115039017304070 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003fde1b0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115041481457666} {CheckpointTs:450115039502270642 MinTableBarrierTs:450115042398961667 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b276fcaf-764a-411e-ae34-b73a9c1ed300 {"id":"b276fcaf-764a-411e-ae34-b73a9c1ed300","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/f356eb82-478a-4ffb-81bf-d0870c09ca41 {"id":"f356eb82-478a-4ffb-81bf-d0870c09ca41","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f5847e2 b276fcaf-764a-411e-ae34-b73a9c1ed300 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f584834 f356eb82-478a-4ffb-81bf-d0870c09ca41 /tidb/cdc/default/default/changefeed/info/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"upstream-id":7374684746182983475,"namespace":"default","changefeed-id":"b51e63e2-45ad-4f8b-b762-ba2a49072db9","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T15:03:37.962726757+08:00","start-ts":450115039017304070,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115041481457666} /tidb/cdc/default/default/changefeed/status/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"checkpoint-ts":450115039502270642,"min-table-barrier-ts":450115042398961667,"admin-job-type":0} /tidb/cdc/default/default/task/position/b276fcaf-764a-411e-ae34-b73a9c1ed300/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f356eb82-478a-4ffb-81bf-d0870c09ca41/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684746182983475 {"id":7374684746182983475,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowe+ grep -q 'failed to get info:' d-cn":null}' + echo ' *** processors info ***: changefeedID: default/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {UpstreamID:7374684746182983475 Namespace:default ID:b51e63e2-45ad-4f8b-b762-ba2a49072db9 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-30 15:03:37.962726757 +0800 CST StartTs:450115039017304070 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003fde1b0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115041481457666} {CheckpointTs:450115039502270642 MinTableBarrierTs:450115042398961667 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b276fcaf-764a-411e-ae34-b73a9c1ed300 {"id":"b276fcaf-764a-411e-ae34-b73a9c1ed300","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/f356eb82-478a-4ffb-81bf-d0870c09ca41 {"id":"f356eb82-478a-4ffb-81bf-d0870c09ca41","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f5847e2 b276fcaf-764a-411e-ae34-b73a9c1ed300 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f584834 f356eb82-478a-4ffb-81bf-d0870c09ca41 /tidb/cdc/default/default/changefeed/info/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"upstream-id":7374684746182983475,"namespace":"default","changefeed-id":"b51e63e2-45ad-4f8b-b762-ba2a49072db9","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T15:03:37.962726757+08:00","start-ts":450115039017304070,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115041481457666} /tidb/cdc/default/default/changefeed/status/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"checkpoint-ts":450115039502270642,"min-table-barrier-ts":450115042398961667,"admin-job-type":0} /tidb/cdc/default/default/task/position/b276fcaf-764a-411e-ae34-b73a9c1ed300/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f356eb82-478a-4ffb-81bf-d0870c09ca41/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684746182983475 {"id":7374684746182983475,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Thu May 30 15:03:42 CST 2024] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.2207122073.out server --log-file /tmp/tidb_cdc_test/move_table/cdc3.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data3 --cluster-id default --addr 127.0.0.1:8302 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 start tidb cluster in /tmp/tidb_cdc_test/changefeed_finish Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/consistent_replicate_storage_file_large_value/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table mark.finish_mark_3 not exists for 9-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:03:45 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {UpstreamID:7374684746182983475 Namespace:default ID:b51e63e2-45ad-4f8b-b762-ba2a49072db9 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-30 15:03:37.962726757 +0800 CST StartTs:450115039017304070 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0034fef30 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115041481457666} {CheckpointTs:450115039816843447 MinTableBarrierTs:450115043464577030 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b276fcaf-764a-411e-ae34-b73a9c1ed300 {"id":"b276fcaf-764a-411e-ae34-b73a9c1ed300","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/cb3da729-3839-43b3-b35f-cd431a82d47a {"id":"cb3da729-3839-43b3-b35f-cd431a82d47a","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/f356eb82-478a-4ffb-81bf-d0870c09ca41 {"id":"f356eb82-478a-4ffb-81bf-d0870c09ca41","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f5847e2 b276fcaf-764a-411e-ae34-b73a9c1ed300 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f584834 f356eb82-478a-4ffb-81bf-d0870c09ca41 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f58486b cb3da729-3839-43b3-b35f-cd431a82d47a /tidb/cdc/default/default/changefeed/info/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"upstream-id":7374684746182983475,"namespace":"default","changefeed-id":"b51e63e2-45ad-4f8b-b762-ba2a49072db9","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T15:03:37.962726757+08:00","start-ts":450115039017304070,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115041481457666} /tidb/cdc/default/default/changefeed/status/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"checkpoint-ts":450115039829950507,"min-table-barrier-ts":450115043464577030,"admin-job-type":0} /tidb/cdc/default/default/task/position/b276fcaf-764a-411e-ae34-b73a9c1ed300/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/cb3da729-3839-43b3-b35f-cd431a82d47a/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f356eb82-478a-4ffb-81bf-d0870c09ca41/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684746182983475 {"id":7374684746182983475,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {UpstreamID:7374684746182983475 Namespace:default ID:b51e63e2-45ad-4f8b-b762-ba2a49072db9 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-30 15:03:37.962726757 +0800 CST StartTs:450115039017304070 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0034fef30 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115041481457666} {CheckpointTs:450115039816843447 MinTableBarrierTs:450115043464577030 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b276fcaf-764a-411e-ae34-b73a9c1ed300 {"id":"b276fcaf-764a-411e-ae34-b73a9c1ed300","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/cb3da729-3839-43b3-b35f-cd431a82d47a {"id":"cb3da729-3839-43b3-b35f-cd431a82d47a","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/f356eb82-478a-4ffb-81bf-d0870c09ca41 {"id":"f356eb82-478a-4ffb-81bf-d0870c09ca41","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f5847e2 b276fcaf-764a-411e-ae34-b73a9c1ed300 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f584834 f356eb82-478a-4ffb-81bf-d0870c09ca41 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f58486b cb3da729-3839-43b3-b35f-cd431a82d47a /tidb/cdc/default/default/changefeed/info/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"upstream-id":7374684746182983475,"namespace":"default","changefeed-id":"b51e63e2-45ad-4f8b-b762-ba2a49072db9","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T15:03:37.962726757+08:00","start-ts":450115039017304070,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115041481457666} /tidb/cdc/default/default/changefeed/status/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"checkpoint-ts":450115039829950507,"min-table-barrier-ts":450115043464577030,"admin-job-type":0} /tidb/cdc/default/default/task/position/b276fcaf-764a-411e-ae34-b73a9c1ed300/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/cb3da729-3839-43b3-b35f-cd431a82d47a/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f356eb82-478a-4ffb-81bf-d0870c09ca41/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684746182983475 {"id":7374684746182983475,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {UpstreamID:7374684746182983475 Namespace:default ID:b51e63e2-45ad-4f8b-b762-ba2a49072db9 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-30 15:03:37.962726757 +0800 CST StartTs:450115039017304070 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0034fef30 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115041481457666} {CheckpointTs:450115039816843447 MinTableBarrierTs:450115043464577030 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b276fcaf-764a-411e-ae34-b73a9c1ed300 {"id":"b276fcaf-764a-411e-ae34-b73a9c1ed300","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/cb3da729-3839-43b3-b35f-cd431a82d47a {"id":"cb3da729-3839-43b3-b35f-cd431a82d47a","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/f356eb82-478a-4ffb-81bf-d0870c09ca41 {"id":"f356eb82-478a-4ffb-81bf-d0870c09ca41","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f5847e2 b276fcaf-764a-411e-ae34-b73a9c1ed300 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f584834 f356eb82-478a-4ffb-81bf-d0870c09ca41 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f58486b cb3da729-3839-43b3-b35f-cd431a82d47a /tidb/cdc/default/default/changefeed/info/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"upstream-id":7374684746182983475,"namespace":"default","changefeed-id":"b51e63e2-45ad-4f8b-b762-ba2a49072db9","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T15:03:37.962726757+08:00","start-ts":450115039017304070,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115041481457666} /tidb/cdc/default/default/changefeed/status/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"checkpoint-ts":450115039829950507,"min-table-barrier-ts":450115043464577030,"admin-job-type":0} /tidb/cdc/default/default/task/position/b276fcaf-764a-411e-ae34-b73a9c1ed300/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/cb3da729-3839-43b3-b35f-cd431a82d47a/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f356eb82-478a-4ffb-81bf-d0870c09ca41/b51e63e2-45ad-4f8b-b762-ba2a49072db9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684746182983475 {"id":7374684746182983475,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x table move_table.usertable exists go: downloading github.com/benbjohnson/clock v1.3.5 go: downloading go.etcd.io/etcd/server/v3 v3.5.12 go: downloading github.com/xdg/scram v1.0.5 go: downloading github.com/IBM/sarama v1.41.2 go: downloading github.com/tinylib/msgp v1.1.6 go: downloading github.com/apache/pulsar-client-go v0.11.0 go: downloading github.com/aws/aws-sdk-go-v2 v1.19.1 go: downloading github.com/KimMachineGun/automemlimit v0.2.4 go: downloading github.com/go-mysql-org/go-mysql v1.7.1-0.20240507075657-2bd4573edde2 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/cakturk/go-netstat v0.0.0-20200220111822-e5b49efee7a5 start tidb cluster in /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2 go: downloading github.com/xdg/stringprep v1.0.3 go: downloading github.com/stretchr/objx v0.5.2 go: downloading github.com/containerd/cgroups v1.0.4 [2024/05/30 15:03:46.147 +08:00] [INFO] [main.go:86] ["running ddl test: 2 addDropColumnDDL"] [2024/05/30 15:03:46.416 +08:00] [INFO] [main.go:220] ["1 insert success: 100"] [2024/05/30 15:03:46.428 +08:00] [INFO] [main.go:220] ["0 insert success: 100"] [2024/05/30 15:03:46.643 +08:00] [INFO] [main.go:234] ["1 delete success: 100"] [2024/05/30 15:03:46.644 +08:00] [INFO] [main.go:220] ["1 insert success: 200"] [2024/05/30 15:03:46.664 +08:00] [INFO] [main.go:234] ["0 delete success: 100"] [2024/05/30 15:03:46.665 +08:00] [INFO] [main.go:220] ["0 insert success: 200"] go: downloading github.com/philhofer/fwd v1.1.1 go: downloading github.com/opencontainers/runtime-spec v1.0.2 go: downloading github.com/godbus/dbus/v5 v5.0.4 go: downloading github.com/cilium/ebpf v0.4.0 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading github.com/siddontang/go v0.0.0-20180604090527-bdc77568d726 go: downloading github.com/siddontang/go-log v0.0.0-20180807004314-8d05993dda07 [2024/05/30 15:03:46.891 +08:00] [INFO] [main.go:220] ["1 insert success: 300"] [2024/05/30 15:03:46.901 +08:00] [INFO] [main.go:220] ["0 insert success: 300"] go: downloading github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 go: downloading github.com/aws/smithy-go v1.13.5 go: downloading github.com/bits-and-blooms/bitset v1.4.0 go: downloading github.com/AthenZ/athenz v1.10.39 go: downloading github.com/pierrec/lz4 v2.6.1+incompatible go: downloading github.com/linkedin/goavro/v2 v2.11.1 go: downloading golang.org/x/mod v0.17.0 go: downloading github.com/99designs/keyring v1.2.1 go: downloading github.com/spaolacci/murmur3 v1.1.0 go: downloading github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 go: downloading github.com/jcmturner/gofork v1.7.6 go: downloading github.com/jcmturner/gokrb5/v8 v8.4.4 go: downloading github.com/eapache/go-resiliency v1.4.0 go: downloading github.com/eapache/queue v1.1.0 go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 table mark.finish_mark_3 not exists for 10-th check, retry later [2024/05/30 15:03:47.120 +08:00] [INFO] [main.go:234] ["1 delete success: 200"] [2024/05/30 15:03:47.122 +08:00] [INFO] [main.go:220] ["1 insert success: 400"] [2024/05/30 15:03:47.133 +08:00] [INFO] [main.go:234] ["0 delete success: 200"] [2024/05/30 15:03:47.134 +08:00] [INFO] [main.go:220] ["0 insert success: 400"] go: downloading github.com/hashicorp/errwrap v1.0.0 go: downloading github.com/dvsekhvalnov/jose2go v1.5.0 go: downloading github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c go: downloading github.com/mtibben/percent v0.2.1 [2024/05/30 15:03:47.357 +08:00] [INFO] [main.go:220] ["1 insert success: 500"] [2024/05/30 15:03:47.381 +08:00] [INFO] [main.go:220] ["0 insert success: 500"] go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0 go: downloading go.etcd.io/bbolt v1.3.9 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 go: downloading go.etcd.io/etcd/pkg/v3 v3.5.12 go: downloading github.com/golang-jwt/jwt/v4 v4.5.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.22.0 go: downloading github.com/jonboulle/clockwork v0.4.0 go: downloading go.opentelemetry.io/otel/sdk v1.22.0 go: downloading github.com/soheilhy/cmux v0.1.5 go: downloading github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 go: downloading go.etcd.io/etcd/client/v2 v2.305.12 go: downloading go.etcd.io/etcd/raft/v3 v3.5.12 go: downloading github.com/jcmturner/dnsutils/v2 v2.0.0 go: downloading github.com/hashicorp/go-uuid v1.0.3 go: downloading github.com/jcmturner/aescts/v2 v2.0.0 go: downloading github.com/jcmturner/rpc/v2 v2.0.3 go: downloading github.com/gorilla/websocket v1.5.1 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 go: downloading go.opentelemetry.io/proto/otlp v1.1.0 go: downloading github.com/cenkalti/backoff/v4 v4.2.1 [2024/05/30 15:03:47.641 +08:00] [INFO] [main.go:234] ["1 delete success: 300"] [2024/05/30 15:03:47.643 +08:00] [INFO] [main.go:220] ["1 insert success: 600"] [2024/05/30 15:03:47.736 +08:00] [INFO] [main.go:234] ["0 delete success: 300"] [2024/05/30 15:03:47.737 +08:00] [INFO] [main.go:220] ["0 insert success: 600"] go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/multi_cdc_cluster/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2024/05/30 15:03:48.043 +08:00] [INFO] [main.go:220] ["1 insert success: 700"] [2024/05/30 15:03:48.057 +08:00] [INFO] [main.go:220] ["0 insert success: 700"] [2024/05/30 15:03:48.277 +08:00] [INFO] [main.go:234] ["1 delete success: 400"] [2024/05/30 15:03:48.279 +08:00] [INFO] [main.go:220] ["1 insert success: 800"] [2024/05/30 15:03:48.290 +08:00] [INFO] [main.go:234] ["0 delete success: 400"] [2024/05/30 15:03:48.291 +08:00] [INFO] [main.go:220] ["0 insert success: 800"] [2024/05/30 15:03:48.504 +08:00] [INFO] [main.go:220] ["1 insert success: 900"] [2024/05/30 15:03:48.513 +08:00] [INFO] [main.go:220] ["0 insert success: 900"] wait process 8090 exit for 1-th time... wait process 8090 exit for 2-th time... wait process 8090 exit for 3-th time... /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (8090) - No such process wait process 8090 exit for 4-th time... process 8090 already exit check_capture_count http://127.0.0.1:2379 0 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Thu May 30 15:03:48 CST 2024] <<<<<< START cdc server in kill_owner_with_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql/MySQLSinkExecDDLDelay=return(true);github.com/pingcap/tiflow/cdc/capture/ownerFlushIntervalInject=return(10)' + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kill_owner_with_ddl.82328234.out server --log-file /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/05/30 15:03:48.728 +08:00] [INFO] [main.go:234] ["0 delete success: 500"] [2024/05/30 15:03:48.729 +08:00] [INFO] [main.go:220] ["0 insert success: 1000"] [2024/05/30 15:03:48.731 +08:00] [INFO] [main.go:234] ["1 delete success: 500"] [2024/05/30 15:03:48.732 +08:00] [INFO] [main.go:220] ["1 insert success: 1000"] table mark.finish_mark_3 not exists for 11-th check, retry later [2024/05/30 15:03:48.940 +08:00] [INFO] [main.go:220] ["0 insert success: 1100"] [2024/05/30 15:03:48.945 +08:00] [INFO] [main.go:220] ["1 insert success: 1100"] [2024/05/30 15:03:49.171 +08:00] [INFO] [main.go:234] ["0 delete success: 600"] [2024/05/30 15:03:49.173 +08:00] [INFO] [main.go:220] ["0 insert success: 1200"] [2024/05/30 15:03:49.383 +08:00] [INFO] [main.go:220] ["0 insert success: 1300"] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:03:49.744 +08:00] [INFO] [main.go:234] ["0 delete success: 700"] [2024/05/30 15:03:49.745 +08:00] [INFO] [main.go:220] ["0 insert success: 1400"] [2024/05/30 15:03:50.084 +08:00] [INFO] [main.go:220] ["0 insert success: 1500"] [2024/05/30 15:03:50.335 +08:00] [INFO] [main.go:234] ["1 delete success: 600"] [2024/05/30 15:03:50.337 +08:00] [INFO] [main.go:220] ["1 insert success: 1200"] [2024/05/30 15:03:50.359 +08:00] [INFO] [main.go:234] ["0 delete success: 800"] [2024/05/30 15:03:50.361 +08:00] [INFO] [main.go:220] ["0 insert success: 1600"] [2024/05/30 15:03:50.615 +08:00] [INFO] [main.go:220] ["1 insert success: 1300"] start tidb cluster in /tmp/tidb_cdc_test/multi_cdc_cluster Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... [2024/05/30 15:03:50.655 +08:00] [INFO] [main.go:220] ["0 insert success: 1700"] table mark.finish_mark_3 not exists for 12-th check, retry later [2024/05/30 15:03:50.880 +08:00] [INFO] [main.go:234] ["1 delete success: 700"] [2024/05/30 15:03:50.881 +08:00] [INFO] [main.go:220] ["1 insert success: 1400"] [2024/05/30 15:03:51.131 +08:00] [INFO] [main.go:220] ["1 insert success: 1500"] Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:03:51 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/743c206c-db17-4f29-b103-9e56a2826b2d {UpstreamID:7374684617514448989 Namespace:default ID:743c206c-db17-4f29-b103-9e56a2826b2d SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-30 15:03:04.811090645 +0800 CST StartTs:450115032747606021 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00399aea0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115032786927619} {CheckpointTs:450115035014889498 MinTableBarrierTs:450115035014889498 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115035014889498, checkpointTs: 450115035014889498, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/eb96efed-7e3a-486c-b3ad-9d8540cbeffd {"id":"eb96efed-7e3a-486c-b3ad-9d8540cbeffd","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ee1f984 eb96efed-7e3a-486c-b3ad-9d8540cbeffd /tidb/cdc/default/default/changefeed/info/743c206c-db17-4f29-b103-9e56a2826b2d {"upstream-id":7374684617514448989,"namespace":"default","changefeed-id":"743c206c-db17-4f29-b103-9e56a2826b2d","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T15:03:04.811090645+08:00","start-ts":450115032747606021,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115032786927619} /tidb/cdc/default/default/changefeed/status/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":450115035014889498,"min-table-barrier-ts":450115035014889498,"admin-job-type":0} /tidb/cdc/default/default/task/position/eb96efed-7e3a-486c-b3ad-9d8540cbeffd/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684617514448989 {"id":7374684617514448989,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/743c206c-db17-4f29-b103-9e56a2826b2d {UpstreamID:7374684617514448989 Namespace:default ID:743c206c-db17-4f29-b103-9e56a2826b2d SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-30 15:03:04.811090645 +0800 CST StartTs:450115032747606021 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00399aea0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115032786927619} {CheckpointTs:450115035014889498 MinTableBarrierTs:450115035014889498 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115035014889498, checkpointTs: 450115035014889498, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/eb96efed-7e3a-486c-b3ad-9d8540cbeffd {"id":"eb96efed-7e3a-486c-b3ad-9d8540cbeffd","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ee1f984 eb96efed-7e3a-486c-b3ad-9d8540cbeffd /tidb/cdc/default/default/changefeed/info/743c206c-db17-4f29-b103-9e56a2826b2d {"upstream-id":7374684617514448989,"namespace":"default","changefeed-id":"743c206c-db17-4f29-b103-9e56a2826b2d","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T15:03:04.811090645+08:00","start-ts":450115032747606021,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115032786927619} /tidb/cdc/default/default/changefeed/status/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":450115035014889498,"min-table-barrier-ts":450115035014889498,"admin-job-type":0} /tidb/cdc/default/default/task/position/eb96efed-7e3a-486c-b3ad-9d8540cbeffd/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684617514448989 + grep -q 'failed to get info:' {"id":7374684617514448989,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/743c206c-db17-4f29-b103-9e56a2826b2d {UpstreamID:7374684617514448989 Namespace:default ID:743c206c-db17-4f29-b103-9e56a2826b2d SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-30 15:03:04.811090645 +0800 CST StartTs:450115032747606021 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00399aea0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115032786927619} {CheckpointTs:450115035014889498 MinTableBarrierTs:450115035014889498 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115035014889498, checkpointTs: 450115035014889498, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/eb96efed-7e3a-486c-b3ad-9d8540cbeffd {"id":"eb96efed-7e3a-486c-b3ad-9d8540cbeffd","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ee1f984 eb96efed-7e3a-486c-b3ad-9d8540cbeffd /tidb/cdc/default/default/changefeed/info/743c206c-db17-4f29-b103-9e56a2826b2d {"upstream-id":7374684617514448989,"namespace":"default","changefeed-id":"743c206c-db17-4f29-b103-9e56a2826b2d","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T15:03:04.811090645+08:00","start-ts":450115032747606021,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115032786927619} /tidb/cdc/default/default/changefeed/status/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":450115035014889498,"min-table-barrier-ts":450115035014889498,"admin-job-type":0} /tidb/cdc/default/default/task/position/eb96efed-7e3a-486c-b3ad-9d8540cbeffd/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684617514448989 + grep -q 'etcd info' {"id":7374684617514448989,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x check_capture_count http://127.0.0.1:2379 1 run task successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... table mark.finish_mark_3 not exists for 13-th check, retry later go: downloading github.com/ardielle/ardielle-go v1.5.2 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213fbbb4000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-nk3lw, pid:4010, start at 2024-05-30 15:03:52.824457779 +0800 CST m=+5.119413187 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:05:52.831 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:03:52.813 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:53:52.813 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213fbbb4000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-nk3lw, pid:4010, start at 2024-05-30 15:03:52.824457779 +0800 CST m=+5.119413187 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:05:52.831 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:03:52.813 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:53:52.813 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213fbd140015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-nk3lw, pid:4081, start at 2024-05-30 15:03:52.945289715 +0800 CST m=+5.190195367 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:05:52.952 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:03:52.952 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:53:52.952 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_finish/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_finish/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } table mark.finish_mark_3 not exists for 14-th check, retry later [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Thu May 30 15:03:56 CST 2024] <<<<<< START cdc server in changefeed_finish case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_finish.53395341.out server --log-file /tmp/tidb_cdc_test/changefeed_finish/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_finish/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/05/30 15:03:54.502 +08:00] [WARN] [diff.go:765] ["rows is not equal"] [table=`test`.`ntest`] [where="((TRUE) AND TRUE)"] [cost=1m34.854589425s] [2024/05/30 15:03:54.505 +08:00] [WARN] [diff.go:383] ["check chunk data not equal"] [chunk="{\"id\":0,\"bounds\":[],\"where\":\"((TRUE) AND TRUE)\",\"args\":null,\"state\":\"failed\"}"] table mark.finish_mark_3 not exists for 15-th check, retry later [2024/05/30 15:03:57.380 +08:00] [INFO] [main.go:86] ["running ddl test: 3 addDropColumnDDL2"] [2024/05/30 15:03:57.642 +08:00] [INFO] [main.go:220] ["1 insert success: 100"] [2024/05/30 15:03:57.659 +08:00] [INFO] [main.go:220] ["0 insert success: 100"] [2024/05/30 15:03:57.855 +08:00] [INFO] [main.go:220] ["1 insert success: 200"] [2024/05/30 15:03:57.857 +08:00] [INFO] [main.go:234] ["1 delete success: 100"] [2024/05/30 15:03:57.879 +08:00] [INFO] [main.go:234] ["0 delete success: 100"] [2024/05/30 15:03:57.880 +08:00] [INFO] [main.go:220] ["0 insert success: 200"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:03:58.073 +08:00] [INFO] [main.go:220] ["1 insert success: 300"] [2024/05/30 15:03:58.114 +08:00] [INFO] [main.go:220] ["0 insert success: 300"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:03:58.302 +08:00] [INFO] [main.go:220] ["1 insert success: 400"] [2024/05/30 15:03:58.304 +08:00] [INFO] [main.go:234] ["1 delete success: 200"] [2024/05/30 15:03:58.337 +08:00] [INFO] [main.go:234] ["0 delete success: 200"] [2024/05/30 15:03:58.338 +08:00] [INFO] [main.go:220] ["0 insert success: 400"] [2024/05/30 15:03:58.496 +08:00] [INFO] [main.go:220] ["1 insert success: 500"] [2024/05/30 15:03:58.536 +08:00] [INFO] [main.go:220] ["0 insert success: 500"] [2024/05/30 15:03:58.694 +08:00] [INFO] [main.go:220] ["1 insert success: 600"] [2024/05/30 15:03:58.696 +08:00] [INFO] [main.go:234] ["1 delete success: 300"] [2024/05/30 15:03:58.733 +08:00] [INFO] [main.go:234] ["0 delete success: 300"] [2024/05/30 15:03:58.734 +08:00] [INFO] [main.go:220] ["0 insert success: 600"] [2024/05/30 15:03:58.893 +08:00] [INFO] [main.go:220] ["1 insert success: 700"] table mark.finish_mark_3 not exists for 16-th check, retry later [2024/05/30 15:03:58.955 +08:00] [INFO] [main.go:220] ["0 insert success: 700"] + (( i++ )) + (( i <= 50 )) [2024/05/30 15:03:59.077 +08:00] [INFO] [main.go:220] ["1 insert success: 800"] ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info [2024/05/30 15:03:59.079 +08:00] [INFO] [main.go:234] ["1 delete success: 400"] * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... [2024/05/30 15:03:59.143 +08:00] [INFO] [main.go:234] ["0 delete success: 400"] * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 [2024/05/30 15:03:59.144 +08:00] [INFO] [main.go:220] ["0 insert success: 800"] > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:03:59 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/729c78fa-7c06-40eb-88fb-f890837ac7b4 {"id":"729c78fa-7c06-40eb-88fb-f890837ac7b4","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84fc710b7 729c78fa-7c06-40eb-88fb-f890837ac7b4 /tidb/cdc/default/default/upstream/7374684867512116127 {"id":7374684867512116127,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/729c78fa-7c06-40eb-88fb-f890837ac7b4 {"id":"729c78fa-7c06-40eb-88fb-f890837ac7b4","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84fc710b7 729c78fa-7c06-40eb-88fb-f890837ac7b4 /tidb/cdc/default/default/upstream/7374684867512116127 {"id":7374684867512116127,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/729c78fa-7c06-40eb-88fb-f890837ac7b4 {"id":"729c78fa-7c06-40eb-88fb-f890837ac7b4","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84fc710b7 729c78fa-7c06-40eb-88fb-f890837ac7b4 /tidb/cdc/default/default/upstream/7374684867512116127 {"id":7374684867512116127,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [2024/05/30 15:03:59.265 +08:00] [INFO] [main.go:220] ["1 insert success: 900"] [2024/05/30 15:03:59.342 +08:00] [INFO] [main.go:220] ["0 insert success: 900"] [2024/05/30 15:03:59.454 +08:00] [INFO] [main.go:220] ["1 insert success: 1000"] [2024/05/30 15:03:59.456 +08:00] [INFO] [main.go:234] ["1 delete success: 500"] [2024/05/30 15:03:59.524 +08:00] [INFO] [main.go:234] ["0 delete success: 500"] [2024/05/30 15:03:59.525 +08:00] [INFO] [main.go:220] ["0 insert success: 1000"] [2024/05/30 15:03:59.659 +08:00] [INFO] [main.go:220] ["1 insert success: 1100"] [2024/05/30 15:03:59.729 +08:00] [INFO] [main.go:220] ["0 insert success: 1100"] [2024/05/30 15:03:59.872 +08:00] [INFO] [main.go:220] ["1 insert success: 1200"] [2024/05/30 15:03:59.875 +08:00] [INFO] [main.go:234] ["1 delete success: 600"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213ffdf8000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3, pid:12847, start at 2024-05-30 15:03:57.068301201 +0800 CST m=+5.554579545 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:05:57.077 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:03:57.054 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:53:57.054 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213ffdf8000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3, pid:12847, start at 2024-05-30 15:03:57.068301201 +0800 CST m=+5.554579545 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:05:57.077 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:03:57.054 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:53:57.054 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213ffed40013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3, pid:12916, start at 2024-05-30 15:03:57.143281776 +0800 CST m=+5.562663716 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:05:57.152 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:03:57.109 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:53:57.109 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2024/05/30 15:03:59.958 +08:00] [INFO] [main.go:234] ["0 delete success: 600"] [2024/05/30 15:03:59.959 +08:00] [INFO] [main.go:220] ["0 insert success: 1200"] [2024/05/30 15:04:00.088 +08:00] [INFO] [main.go:220] ["1 insert success: 1300"] [2024/05/30 15:04:00.173 +08:00] [INFO] [main.go:220] ["0 insert success: 1300"] [2024/05/30 15:04:00.318 +08:00] [INFO] [main.go:220] ["1 insert success: 1400"] [2024/05/30 15:04:00.320 +08:00] [INFO] [main.go:234] ["1 delete success: 700"] [2024/05/30 15:04:00.388 +08:00] [INFO] [main.go:234] ["0 delete success: 700"] [2024/05/30 15:04:00.389 +08:00] [INFO] [main.go:220] ["0 insert success: 1400"] [2024/05/30 15:04:00.525 +08:00] [INFO] [main.go:220] ["1 insert success: 1500"] [2024/05/30 15:04:00.592 +08:00] [INFO] [main.go:220] ["0 insert success: 1500"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 1-th time, retry later [2024/05/30 15:04:00.748 +08:00] [INFO] [main.go:220] ["1 insert success: 1600"] [2024/05/30 15:04:00.750 +08:00] [INFO] [main.go:234] ["1 delete success: 800"] [2024/05/30 15:04:00.800 +08:00] [INFO] [main.go:234] ["0 delete success: 800"] [2024/05/30 15:04:00.801 +08:00] [INFO] [main.go:220] ["0 insert success: 1600"] table mark.finish_mark_3 not exists for 17-th check, retry later [2024/05/30 15:04:00.563 +08:00] [INFO] [dailytest.go:68] ["test pass!!!"] [2024/05/30 15:04:00.967 +08:00] [INFO] [main.go:220] ["1 insert success: 1700"] [2024/05/30 15:04:01.024 +08:00] [INFO] [main.go:220] ["0 insert success: 1700"] [2024/05/30 15:04:01.211 +08:00] [INFO] [main.go:220] ["1 insert success: 1800"] [2024/05/30 15:04:01.216 +08:00] [INFO] [main.go:234] ["1 delete success: 900"] [2024/05/30 15:04:01.265 +08:00] [INFO] [main.go:234] ["0 delete success: 900"] [2024/05/30 15:04:01.266 +08:00] [INFO] [main.go:220] ["0 insert success: 1800"] [2024/05/30 15:04:01.434 +08:00] [INFO] [main.go:220] ["1 insert success: 1900"] wait process cdc.test exit for 1-th time... [2024/05/30 15:04:01.619 +08:00] [INFO] [main.go:220] ["0 insert success: 1900"] wait process 8237 exit for 1-th time... wait process 8237 exit for 2-th time... /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (8237) - No such process wait process 8237 exit for 3-th time... process 8237 already exit check_capture_count http://127.0.0.1:2379 0 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Thu May 30 15:04:01 CST 2024] <<<<<< START cdc server in kill_owner_with_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kill_owner_with_ddl.83718373.out server --log-file /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 2-th time... [2024/05/30 15:04:01.742 +08:00] [INFO] [main.go:220] ["1 insert success: 2000"] [2024/05/30 15:04:01.744 +08:00] [INFO] [main.go:234] ["1 delete success: 1000"] [2024/05/30 15:04:01.957 +08:00] [INFO] [main.go:234] ["0 delete success: 1000"] [2024/05/30 15:04:01.958 +08:00] [INFO] [main.go:220] ["0 insert success: 2000"] [2024/05/30 15:04:02.026 +08:00] [INFO] [main.go:220] ["1 insert success: 2100"] [2024/05/30 15:04:02.175 +08:00] [INFO] [main.go:220] ["0 insert success: 2100"] [2024/05/30 15:04:02.250 +08:00] [INFO] [main.go:220] ["1 insert success: 2200"] [2024/05/30 15:04:02.253 +08:00] [INFO] [main.go:234] ["1 delete success: 1100"] wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 15:04:02 CST 2024] <<<<<< run test case cdc success! >>>>>> [Thu May 30 15:04:01 CST 2024] <<<<<< START cdc server in consistent_replicate_storage_file_large_value case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_storage_file_large_value.1417814180.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214041400010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-bcfm9, pid:10671, start at 2024-05-30 15:04:01.376857422 +0800 CST m=+5.240823641 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:06:01.384 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:04:01.360 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:54:01.360 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_3 not exists for 18-th check, retry later check diff failed 2-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214041400010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-bcfm9, pid:10671, start at 2024-05-30 15:04:01.376857422 +0800 CST m=+5.240823641 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:06:01.384 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:04:01.360 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:54:01.360 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214042bc0004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-bcfm9, pid:10748, start at 2024-05-30 15:04:01.456439883 +0800 CST m=+5.234415992 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:06:01.464 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:04:01.455 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:54:01.455 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_cdc_cluster/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_cdc_cluster/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/multi_cdc_cluster/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_cdc_cluster/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/multi_cdc_cluster/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table mark.finish_mark_3 not exists for 19-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:04:04 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/743c206c-db17-4f29-b103-9e56a2826b2d {UpstreamID:7374684617514448989 Namespace:default ID:743c206c-db17-4f29-b103-9e56a2826b2d SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-30 15:03:04.811090645 +0800 CST StartTs:450115032747606021 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00401a5a0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115032786927619} {CheckpointTs:450115035014889498 MinTableBarrierTs:450115035159330826 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115035014889498, checkpointTs: 450115035014889498, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450115035014889498, checkpointTs: 450115035014889498, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e3c23a17-e609-4859-b19d-802c62067602 {"id":"e3c23a17-e609-4859-b19d-802c62067602","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ee1f9fd e3c23a17-e609-4859-b19d-802c62067602 /tidb/cdc/default/default/changefeed/info/743c206c-db17-4f29-b103-9e56a2826b2d {"upstream-id":7374684617514448989,"namespace":"default","changefeed-id":"743c206c-db17-4f29-b103-9e56a2826b2d","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T15:03:04.811090645+08:00","start-ts":450115032747606021,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115032786927619} /tidb/cdc/default/default/changefeed/status/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":450115035014889498,"min-table-barrier-ts":450115035159330826,"admin-job-type":0} /tidb/cdc/default/default/task/position/e3c23a17-e609-4859-b19d-802c62067602/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684617514448989 {"id":7374684617514448989,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/743c206c-db17-4f29-b103-9e56a2826b2d {UpstreamID:7374684617514448989 Namespace:default ID:743c206c-db17-4f29-b103-9e56a2826b2d SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-30 15:03:04.811090645 +0800 CST StartTs:450115032747606021 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00401a5a0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115032786927619} {CheckpointTs:450115035014889498 MinTableBarrierTs:450115035159330826 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115035014889498, checkpointTs: 450115035014889498, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450115035014889498, checkpointTs: 450115035014889498, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e3c23a17-e609-4859-b19d-802c62067602 {"id":"e3c23a17-e609-4859-b19d-802c62067602","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ee1f9fd e3c23a17-e609-4859-b19d-802c62067602 /tidb/cdc/default/default/changefeed/info/743c206c-db17-4f29-b103-9e56a2826b2d {"upstream-id":7374684617514448989,"namespace":"default","changefeed-id":"743c206c-db17-4f29-b103-9e56a2826b2d","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T15:03:04.811090645+08:00","start-ts":450115032747606021,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115032786927619} /tidb/cdc/default/default/changefeed/status/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":450115035014889498,"min-table-barrier-ts":450115035159330826,"admin-job-type":0} /tidb/cdc/default/default/task/position/e3c23a17-e609-4859-b19d-802c62067602/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684617514448989 {"id":7374684617514448989,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/743c206c-db17-4f29-b103-9e56a2826b2d {UpstreamID:7374684617514448989 Namespace:default ID:743c206c-db17-4f29-b103-9e56a2826b2d SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-30 15:03:04.811090645 +0800 CST StartTs:450115032747606021 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00401a5a0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115032786927619} {CheckpointTs:450115035014889498 MinTableBarrierTs:450115035159330826 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115035014889498, checkpointTs: 450115035014889498, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450115035014889498, checkpointTs: 450115035014889498, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e3c23a17-e609-4859-b19d-802c62067602 {"id":"e3c23a17-e609-4859-b19d-802c62067602","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ee1f9fd e3c23a17-e609-4859-b19d-802c62067602 /tidb/cdc/default/default/changefeed/info/743c206c-db17-4f29-b103-9e56a2826b2d {"upstream-id":7374684617514448989,"namespace":"default","changefeed-id":"743c206c-db17-4f29-b103-9e56a2826b2d","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-30T15:03:04.811090645+08:00","start-ts":450115032747606021,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115032786927619} /tidb/cdc/default/default/changefeed/status/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":450115035014889498,"min-table-barrier-ts":450115035159330826,"admin-job-type":0} /tidb/cdc/default/default/task/position/e3c23a17-e609-4859-b19d-802c62067602/743c206c-db17-4f29-b103-9e56a2826b2d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684617514448989 {"id":7374684617514448989,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x check_capture_count http://127.0.0.1:2379 1 run task successfully table kill_owner_with_ddl.t1 exists table kill_owner_with_ddl.t2 exists table kill_owner_with_ddl.t3 not exists for 1-th check, retry later check diff successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:04:06 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7d7ed218-3b3b-4d2c-a3e9-7364397fe6b4 {"id":"7d7ed218-3b3b-4d2c-a3e9-7364397fe6b4","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84fd3ebbe 7d7ed218-3b3b-4d2c-a3e9-7364397fe6b4 /tidb/cdc/default/default/upstream/7374684882919469615 {"id":7374684882919469615,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7d7ed218-3b3b-4d2c-a3e9-7364397fe6b4 {"id":"7d7ed218-3b3b-4d2c-a3e9-7364397fe6b4","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84fd3ebbe 7d7ed218-3b3b-4d2c-a3e9-7364397fe6b4 /tidb/cdc/default/default/upstream/7374684882919469615 {"id":7374684882919469615,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7d7ed218-3b3b-4d2c-a3e9-7364397fe6b4 {"id":"7d7ed218-3b3b-4d2c-a3e9-7364397fe6b4","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84fd3ebbe 7d7ed218-3b3b-4d2c-a3e9-7364397fe6b4 /tidb/cdc/default/default/upstream/7374684882919469615 {"id":7374684882919469615,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_cdc_cluster.cli.11996.out cli tso query --pd=http://127.0.0.1:2379 table kill_owner_with_ddl.t3 exists check diff failed 1-th time, retry later table mark.finish_mark_3 not exists for 20-th check, retry later ***************** properties ***************** "mysql.host"="127.0.0.1" "mysql.port"="4000" "fieldlength"="15000" "insertproportion"="0" "scanproportion"="0" "workload"="core" "readallfields"="true" "mysql.db"="consistent_replicate_storage_file_large_value" "fieldcount"="1000" "requestdistribution"="uniform" "recordcount"="100" "threadcount"="10" "readproportion"="0" "updateproportion"="0" "dotransactions"="false" "operationcount"="0" "fieldlengthdistribution"="constant" "mysql.user"="root" ********************************************** + set +x + tso='450115049032515585 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115049032515585 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 15:04:08 CST 2024] <<<<<< START cdc server in multi_cdc_cluster case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_cdc_cluster.1204212044.out server --log-file /tmp/tidb_cdc_test/multi_cdc_cluster/cdcmult_cdc.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_cdc_cluster/cdc_datamult_cdc.server1 --cluster-id test1 --addr 127.0.0.1:8300 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/05/30 15:04:08.285 +08:00] [INFO] [main.go:86] ["running ddl test: 4 modifyColumnDDL"] [2024/05/30 15:04:08.527 +08:00] [INFO] [main.go:220] ["1 insert success: 100"] [2024/05/30 15:04:08.530 +08:00] [INFO] [main.go:220] ["0 insert success: 100"] [2024/05/30 15:04:08.737 +08:00] [INFO] [main.go:220] ["1 insert success: 200"] [2024/05/30 15:04:08.739 +08:00] [INFO] [main.go:234] ["1 delete success: 100"] [2024/05/30 15:04:08.741 +08:00] [INFO] [main.go:234] ["0 delete success: 100"] [2024/05/30 15:04:08.742 +08:00] [INFO] [main.go:220] ["0 insert success: 200"] [2024/05/30 15:04:08.989 +08:00] [INFO] [main.go:220] ["1 insert success: 300"] [2024/05/30 15:04:08.992 +08:00] [INFO] [main.go:220] ["0 insert success: 300"] [2024/05/30 15:04:09.181 +08:00] [INFO] [main.go:220] ["1 insert success: 400"] [2024/05/30 15:04:09.183 +08:00] [INFO] [main.go:234] ["1 delete success: 200"] [2024/05/30 15:04:09.190 +08:00] [INFO] [main.go:234] ["0 delete success: 200"] [2024/05/30 15:04:09.191 +08:00] [INFO] [main.go:220] ["0 insert success: 400"] table mark.finish_mark_3 not exists for 21-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... [2024/05/30 15:04:09.379 +08:00] [INFO] [main.go:220] ["1 insert success: 500"] [2024/05/30 15:04:09.399 +08:00] [INFO] [main.go:220] ["0 insert success: 500"] [2024/05/30 15:04:09.585 +08:00] [INFO] [main.go:220] ["1 insert success: 600"] [2024/05/30 15:04:09.587 +08:00] [INFO] [main.go:234] ["1 delete success: 300"] [2024/05/30 15:04:09.597 +08:00] [INFO] [main.go:234] ["0 delete success: 300"] [2024/05/30 15:04:09.598 +08:00] [INFO] [main.go:220] ["0 insert success: 600"] [2024/05/30 15:04:09.797 +08:00] [INFO] [main.go:220] ["1 insert success: 700"] [2024/05/30 15:04:09.812 +08:00] [INFO] [main.go:220] ["0 insert success: 700"] [2024/05/30 15:04:10.001 +08:00] [INFO] [main.go:220] ["1 insert success: 800"] [2024/05/30 15:04:10.003 +08:00] [INFO] [main.go:234] ["1 delete success: 400"] wait process cdc.test exit for 2-th time... [2024/05/30 15:04:10.014 +08:00] [INFO] [main.go:234] ["0 delete success: 400"] [2024/05/30 15:04:10.015 +08:00] [INFO] [main.go:220] ["0 insert success: 800"] [2024/05/30 15:04:10.197 +08:00] [INFO] [main.go:220] ["1 insert success: 900"] [2024/05/30 15:04:10.212 +08:00] [INFO] [main.go:220] ["0 insert success: 900"] [2024/05/30 15:04:10.408 +08:00] [INFO] [main.go:220] ["1 insert success: 1000"] [2024/05/30 15:04:10.410 +08:00] [INFO] [main.go:234] ["1 delete success: 500"] [2024/05/30 15:04:10.424 +08:00] [INFO] [main.go:234] ["0 delete success: 500"] [2024/05/30 15:04:10.425 +08:00] [INFO] [main.go:220] ["0 insert success: 1000"] wait process cdc.test exit for 3-th time... [2024/05/30 15:04:10.753 +08:00] [INFO] [main.go:220] ["1 insert success: 1100"] [2024/05/30 15:04:10.845 +08:00] [INFO] [main.go:220] ["0 insert success: 1100"] wait process cdc.test exit for 4-th time... table mark.finish_mark_3 not exists for 22-th check, retry later [2024/05/30 15:04:11.046 +08:00] [INFO] [main.go:220] ["1 insert success: 1200"] [2024/05/30 15:04:11.049 +08:00] [INFO] [main.go:234] ["1 delete success: 600"] [2024/05/30 15:04:11.070 +08:00] [INFO] [main.go:234] ["0 delete success: 600"] [2024/05/30 15:04:11.071 +08:00] [INFO] [main.go:220] ["0 insert success: 1200"] [2024/05/30 15:04:11.256 +08:00] [INFO] [main.go:220] ["1 insert success: 1300"] [2024/05/30 15:04:11.281 +08:00] [INFO] [main.go:220] ["0 insert success: 1300"] [2024/05/30 15:04:11.454 +08:00] [INFO] [main.go:220] ["1 insert success: 1400"] [2024/05/30 15:04:11.457 +08:00] [INFO] [main.go:234] ["1 delete success: 700"] [2024/05/30 15:04:11.527 +08:00] [INFO] [main.go:234] ["0 delete success: 700"] [2024/05/30 15:04:11.529 +08:00] [INFO] [main.go:220] ["0 insert success: 1400"] cdc.test: no process found wait process cdc.test exit for 5-th time... process cdc.test already exit [Thu May 30 15:04:11 CST 2024] <<<<<< run test case kill_owner_with_ddl success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:04:11 GMT < Content-Length: 605 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/test1/__cdc_meta__/capture/db857648-f55c-4970-9744-57b48ca0f2bc {"id":"db857648-f55c-4970-9744-57b48ca0f2bc","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/test1/__cdc_meta__/meta/meta-version 1 /tidb/cdc/test1/__cdc_meta__/owner/22318fc84fe5e8df db857648-f55c-4970-9744-57b48ca0f2bc /tidb/cdc/test1/default/upstream/7374684898893930088 {"id":7374684898893930088,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/test1/__cdc_meta__/capture/db857648-f55c-4970-9744-57b48ca0f2bc {"id":"db857648-f55c-4970-9744-57b48ca0f2bc","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/test1/__cdc_meta__/meta/meta-version 1 /tidb/cdc/test1/__cdc_meta__/owner/22318fc84fe5e8df db857648-f55c-4970-9744-57b48ca0f2bc /tidb/cdc/test1/default/upstream/7374684898893930088 {"id":7374684898893930088,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/test1/__cdc_meta__/capture/db857648-f55c-4970-9744-57b48ca0f2bc {"id":"db857648-f55c-4970-9744-57b48ca0f2bc","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/test1/__cdc_meta__/meta/meta-version 1 /tidb/cdc/test1/__cdc_meta__/owner/22318fc84fe5e8df db857648-f55c-4970-9744-57b48ca0f2bc /tidb/cdc/test1/default/upstream/7374684898893930088 {"id":7374684898893930088,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Thu May 30 15:04:11 CST 2024] <<<<<< START cdc server in multi_cdc_cluster case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_cdc_cluster.1209012092.out server --log-file /tmp/tidb_cdc_test/multi_cdc_cluster/cdcmult_cdc.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_cdc_cluster/cdc_datamult_cdc.server2 --cluster-id test2 --addr 127.0.0.1:8301 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/05/30 15:04:11.835 +08:00] [INFO] [main.go:220] ["1 insert success: 1500"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/split_region/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table mark.finish_mark_3 not exists for 23-th check, retry later [2024/05/30 15:04:12.770 +08:00] [INFO] [main.go:61] ["table mover started"] [2024/05/30 15:04:12.773 +08:00] [INFO] [main.go:166] ["new cluster initialized"] [2024/05/30 15:04:12.773 +08:00] [DEBUG] [main.go:192] ["retrieved owner ID"] [ownerID=b276fcaf-764a-411e-ae34-b73a9c1ed300] [2024/05/30 15:04:12.774 +08:00] [DEBUG] [main.go:199] ["retrieved owner addr"] [ownerAddr=127.0.0.1:8300] [2024/05/30 15:04:12.774 +08:00] [DEBUG] [main.go:210] ["retrieved changefeeds"] [changefeedsError="json: unsupported type: map[model.ChangeFeedID]*mvccpb.KeyValue"] [2024/05/30 15:04:12.960 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=b51e63e2-45ad-4f8b-b762-ba2a49072db9] [captureID=b276fcaf-764a-411e-ae34-b73a9c1ed300] [processorDetail="{\"table_ids\":[104,106]}"] [2024/05/30 15:04:13.160 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=b51e63e2-45ad-4f8b-b762-ba2a49072db9] [captureID=cb3da729-3839-43b3-b35f-cd431a82d47a] [processorDetail="{\"table_ids\":[]}"] [2024/05/30 15:04:13.360 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=b51e63e2-45ad-4f8b-b762-ba2a49072db9] [captureID=f356eb82-478a-4ffb-81bf-d0870c09ca41] [processorDetail="{\"table_ids\":[]}"] [2024/05/30 15:04:13.360 +08:00] [INFO] [main.go:75] ["task status"] [status="{\"b276fcaf-764a-411e-ae34-b73a9c1ed300\":[{\"ID\":104,\"Changefeed\":\"b51e63e2-45ad-4f8b-b762-ba2a49072db9\"},{\"ID\":106,\"Changefeed\":\"b51e63e2-45ad-4f8b-b762-ba2a49072db9\"}],\"cb3da729-3839-43b3-b35f-cd431a82d47a\":[],\"f356eb82-478a-4ffb-81bf-d0870c09ca41\":[]}"] [2024/05/30 15:04:13.360 +08:00] [DEBUG] [main.go:288] ["preparing HTTP API call to owner"] [formStr="cf-id=b51e63e2-45ad-4f8b-b762-ba2a49072db9&target-cp-id=f356eb82-478a-4ffb-81bf-d0870c09ca41&table-id=104"] [2024/05/30 15:04:13.409 +08:00] [INFO] [main.go:180] ["moved table successful"] [tableID=104] [2024/05/30 15:04:13.409 +08:00] [DEBUG] [main.go:288] ["preparing HTTP API call to owner"] [formStr="cf-id=b51e63e2-45ad-4f8b-b762-ba2a49072db9&target-cp-id=f356eb82-478a-4ffb-81bf-d0870c09ca41&table-id=106"] [2024/05/30 15:04:13.462 +08:00] [INFO] [main.go:180] ["moved table successful"] [tableID=106] [2024/05/30 15:04:13.462 +08:00] [INFO] [main.go:114] ["all tables are moved"] [sourceCapture=b276fcaf-764a-411e-ae34-b73a9c1ed300] [targetCapture=f356eb82-478a-4ffb-81bf-d0870c09ca41] table move_table.check1 exists check diff successfully table mark.finish_mark_3 not exists for 24-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:04:14 GMT < Content-Length: 605 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/test2/__cdc_meta__/capture/d545bf87-d04c-44fe-8525-7f3dac7def74 {"id":"d545bf87-d04c-44fe-8525-7f3dac7def74","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/test2/__cdc_meta__/meta/meta-version 1 /tidb/cdc/test2/__cdc_meta__/owner/22318fc84fe5e901 d545bf87-d04c-44fe-8525-7f3dac7def74 /tidb/cdc/test2/default/upstream/7374684898893930088 {"id":7374684898893930088,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/test2/__cdc_meta__/capture/d545bf87-d04c-44fe-8525-7f3dac7def74 {"id":"d545bf87-d04c-44fe-8525-7f3dac7def74","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/test2/__cdc_meta__/meta/meta-version 1 /tidb/cdc/test2/__cdc_meta__/owner/22318fc84fe5e901 d545bf87-d04c-44fe-8525-7f3dac7def74 /tidb/cdc/test2/default/upstream/7374684898893930088 {"id":7374684898893930088,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/test2/__cdc_meta__/capture/d545bf87-d04c-44fe-8525-7f3dac7def74 {"id":"d545bf87-d04c-44fe-8525-7f3dac7def74","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/test2/__cdc_meta__/meta/meta-version 1 /tidb/cdc/test2/__cdc_meta__/owner/22318fc84fe5e901 d545bf87-d04c-44fe-8525-7f3dac7def74 /tidb/cdc/test2/default/upstream/7374684898893930088 {"id":7374684898893930088,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_cdc_cluster.cli.12133.out cli changefeed create --start-ts=450115049032515585 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --server http://127.0.0.1:8300 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/multi_cdc_cluster/conf/changefeed1.toml Create changefeed successfully! ID: 6e6e2eed-1f2a-4782-8892-86651ea82c76 Info: {"upstream_id":7374684898893930088,"namespace":"default","id":"6e6e2eed-1f2a-4782-8892-86651ea82c76","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T15:04:15.05514243+08:00","start_ts":450115049032515585,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["test.multi_cdc1"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115049032515585,"checkpoint_ts":450115049032515585,"checkpoint_time":"2024-05-30 15:04:06.761"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... start tidb cluster in /tmp/tidb_cdc_test/split_region Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_cdc_cluster.cli.12172.out cli changefeed create --start-ts=450115049032515585 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --server http://127.0.0.1:8301 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/multi_cdc_cluster/conf/changefeed2.toml table mark.finish_mark_3 not exists for 25-th check, retry later Create changefeed successfully! ID: 50c79a9d-e494-4ffa-bfbf-21dc12eb2f30 Info: {"upstream_id":7374684898893930088,"namespace":"default","id":"50c79a9d-e494-4ffa-bfbf-21dc12eb2f30","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T15:04:16.924880618+08:00","start_ts":450115049032515585,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["test.multi_cdc2"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115049032515585,"checkpoint_ts":450115049032515585,"checkpoint_time":"2024-05-30 15:04:06.761"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... INSERT_ERROR - Takes(s): 8.9, Count: 70, OPS: 7.9, Avg(us): 708039, Min(us): 200068, Max(us): 2931761, 95th(us): 0, 99th(us): 0 + set +x table test.multi_cdc1 exists table test.multi_cdc2 exists check diff failed 1-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table mark.finish_mark_3 not exists for 26-th check, retry later check diff successfully Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Run finished, takes 12.897250828s INSERT_ERROR - Takes(s): 11.8, Count: 100, OPS: 8.5, Avg(us): 617236, Min(us): 200068, Max(us): 2931761, 95th(us): 0, 99th(us): 0 wait process cdc.test exit for 1-th time... table mark.finish_mark_3 not exists for 27-th check, retry later [2024/05/30 15:04:20.614 +08:00] [INFO] [main.go:61] ["table mover started"] [2024/05/30 15:04:20.616 +08:00] [INFO] [main.go:166] ["new cluster initialized"] [2024/05/30 15:04:20.617 +08:00] [DEBUG] [main.go:192] ["retrieved owner ID"] [ownerID=b276fcaf-764a-411e-ae34-b73a9c1ed300] [2024/05/30 15:04:20.617 +08:00] [DEBUG] [main.go:199] ["retrieved owner addr"] [ownerAddr=127.0.0.1:8300] [2024/05/30 15:04:20.617 +08:00] [DEBUG] [main.go:210] ["retrieved changefeeds"] [changefeedsError="json: unsupported type: map[model.ChangeFeedID]*mvccpb.KeyValue"] [2024/05/30 15:04:20.809 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=b51e63e2-45ad-4f8b-b762-ba2a49072db9] [captureID=b276fcaf-764a-411e-ae34-b73a9c1ed300] [processorDetail="{\"table_ids\":[108]}"] [2024/05/30 15:04:21.010 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=b51e63e2-45ad-4f8b-b762-ba2a49072db9] [captureID=cb3da729-3839-43b3-b35f-cd431a82d47a] [processorDetail="{\"table_ids\":[]}"] [2024/05/30 15:04:21.209 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=b51e63e2-45ad-4f8b-b762-ba2a49072db9] [captureID=f356eb82-478a-4ffb-81bf-d0870c09ca41] [processorDetail="{\"table_ids\":[106]}"] [2024/05/30 15:04:21.209 +08:00] [INFO] [main.go:75] ["task status"] [status="{\"b276fcaf-764a-411e-ae34-b73a9c1ed300\":[{\"ID\":108,\"Changefeed\":\"b51e63e2-45ad-4f8b-b762-ba2a49072db9\"}],\"cb3da729-3839-43b3-b35f-cd431a82d47a\":[],\"f356eb82-478a-4ffb-81bf-d0870c09ca41\":[{\"ID\":106,\"Changefeed\":\"b51e63e2-45ad-4f8b-b762-ba2a49072db9\"}]}"] [2024/05/30 15:04:21.209 +08:00] [DEBUG] [main.go:288] ["preparing HTTP API call to owner"] [formStr="cf-id=b51e63e2-45ad-4f8b-b762-ba2a49072db9&target-cp-id=f356eb82-478a-4ffb-81bf-d0870c09ca41&table-id=108"] [2024/05/30 15:04:21.260 +08:00] [INFO] [main.go:180] ["moved table successful"] [tableID=108] [2024/05/30 15:04:21.260 +08:00] [INFO] [main.go:114] ["all tables are moved"] [sourceCapture=b276fcaf-764a-411e-ae34-b73a9c1ed300] [targetCapture=f356eb82-478a-4ffb-81bf-d0870c09ca41] check diff successfully table move_table.check2 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/owner_remove_table_error/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... wait process cdc.test exit for 2-th time... [2024/05/30 15:04:20.536 +08:00] [INFO] [main.go:86] ["running ddl test: 5 addDropIndexDDL"] [2024/05/30 15:04:20.885 +08:00] [INFO] [main.go:220] ["1 insert success: 100"] [2024/05/30 15:04:21.092 +08:00] [INFO] [main.go:220] ["1 insert success: 200"] [2024/05/30 15:04:21.094 +08:00] [INFO] [main.go:234] ["1 delete success: 100"] [2024/05/30 15:04:21.259 +08:00] [INFO] [main.go:220] ["1 insert success: 300"] [2024/05/30 15:04:21.444 +08:00] [INFO] [main.go:220] ["1 insert success: 400"] [2024/05/30 15:04:21.446 +08:00] [INFO] [main.go:234] ["1 delete success: 200"] [2024/05/30 15:04:21.630 +08:00] [INFO] [main.go:220] ["1 insert success: 500"] [2024/05/30 15:04:21.826 +08:00] [INFO] [main.go:220] ["1 insert success: 600"] [2024/05/30 15:04:21.828 +08:00] [INFO] [main.go:234] ["1 delete success: 300"] [2024/05/30 15:04:21.889 +08:00] [INFO] [main.go:220] ["0 insert success: 100"] [2024/05/30 15:04:22.028 +08:00] [INFO] [main.go:220] ["1 insert success: 700"] [2024/05/30 15:04:22.215 +08:00] [INFO] [main.go:220] ["1 insert success: 800"] [2024/05/30 15:04:22.217 +08:00] [INFO] [main.go:234] ["1 delete success: 400"] wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 15:04:22 CST 2024] <<<<<< run test case multi_cdc_cluster success! >>>>>> [2024/05/30 15:04:22.402 +08:00] [INFO] [main.go:220] ["1 insert success: 900"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:04:22.591 +08:00] [INFO] [main.go:220] ["1 insert success: 1000"] [2024/05/30 15:04:22.593 +08:00] [INFO] [main.go:234] ["1 delete success: 500"] [2024/05/30 15:04:22.790 +08:00] [INFO] [main.go:220] ["1 insert success: 1100"] [2024/05/30 15:04:22.999 +08:00] [INFO] [main.go:220] ["1 insert success: 1200"] [2024/05/30 15:04:23.002 +08:00] [INFO] [main.go:234] ["1 delete success: 600"] table consistent_replicate_storage_file_large_value.usertable exists table consistent_replicate_storage_file_large_value.check1 not exists for 1-th check, retry later table mark.finish_mark_3 not exists for 28-th check, retry later [2024/05/30 15:04:23.098 +08:00] [INFO] [main.go:234] ["0 delete success: 100"] [2024/05/30 15:04:23.099 +08:00] [INFO] [main.go:220] ["0 insert success: 200"] [2024/05/30 15:04:23.217 +08:00] [INFO] [main.go:220] ["1 insert success: 1300"] [2024/05/30 15:04:23.343 +08:00] [INFO] [main.go:220] ["0 insert success: 300"] table move_table.check2 exists [2024/05/30 15:04:23.632 +08:00] [INFO] [main.go:220] ["1 insert success: 1400"] [2024/05/30 15:04:23.641 +08:00] [INFO] [main.go:234] ["1 delete success: 700"] [2024/05/30 15:04:23.715 +08:00] [INFO] [main.go:234] ["0 delete success: 200"] [2024/05/30 15:04:23.717 +08:00] [INFO] [main.go:220] ["0 insert success: 400"] check diff successfully [2024/05/30 15:04:23.976 +08:00] [INFO] [main.go:220] ["1 insert success: 1500"] start tidb cluster in /tmp/tidb_cdc_test/owner_remove_table_error Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... [2024/05/30 15:04:24.083 +08:00] [INFO] [main.go:220] ["0 insert success: 500"] wait process cdc.test exit for 1-th time... [2024/05/30 15:04:24.354 +08:00] [INFO] [main.go:220] ["1 insert success: 1600"] [2024/05/30 15:04:24.357 +08:00] [INFO] [main.go:234] ["1 delete success: 800"] [2024/05/30 15:04:24.452 +08:00] [INFO] [main.go:234] ["0 delete success: 300"] [2024/05/30 15:04:24.453 +08:00] [INFO] [main.go:220] ["0 insert success: 600"] table mark.finish_mark_3 not exists for 29-th check, retry later [2024/05/30 15:04:24.855 +08:00] [INFO] [main.go:220] ["1 insert success: 1700"] [2024/05/30 15:04:24.972 +08:00] [INFO] [main.go:220] ["0 insert success: 700"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 15:04:25.234 +08:00] [INFO] [main.go:220] ["1 insert success: 1800"] [2024/05/30 15:04:25.240 +08:00] [INFO] [main.go:234] ["1 delete success: 900"] table consistent_replicate_storage_file_large_value.check1 exists table consistent_replicate_storage_file_large_value.t2 not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 15:04:26 CST 2024] <<<<<< run test case move_table success! >>>>>> table mark.finish_mark_3 not exists for 30-th check, retry later Verifying downstream PD is started... table consistent_replicate_storage_file_large_value.t2 not exists for 2-th check, retry later Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_storage_file_large_value.t2 not exists for 3-th check, retry later table mark.finish_mark_3 not exists for 31-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2141d1a00010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l00sh, pid:20541, start at 2024-05-30 15:04:27.013446632 +0800 CST m=+6.565215130 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:06:27.024 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:04:26.984 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:54:26.984 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2141d1a00010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l00sh, pid:20541, start at 2024-05-30 15:04:27.013446632 +0800 CST m=+6.565215130 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:06:27.024 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:04:26.984 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:54:26.984 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2141f4180013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l00sh, pid:20611, start at 2024-05-30 15:04:29.232332915 +0800 CST m=+8.709534629 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:06:29.243 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:04:29.240 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:54:29.240 +0800 All versions after safe point can be accessed. (DO NOT EDIT) table consistent_replicate_storage_file_large_value.t2 not exists for 4-th check, retry later table mark.finish_mark_3 not exists for 32-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } table consistent_replicate_storage_file_large_value.t2 not exists for 5-th check, retry later [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table mark.finish_mark_3 not exists for 33-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/split_region/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/split_region/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/split_region/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/split_region/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/split_region/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table mark.finish_mark_3 not exists for 34-th check, retry later [2024/05/30 15:04:33.529 +08:00] [INFO] [main.go:78] ["runDDLTest take 1m8.850603583s"] table mark.finish_mark_0 exists table mark.finish_mark_1 exists table mark.finish_mark_2 exists table mark.finish_mark_3 exists table mark.finish_mark_4 exists table mark.finish_mark not exists for 1-th check, retry later table consistent_replicate_storage_file_large_value.t2 not exists for 6-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214249080017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-0vnff, pid:9479, start at 2024-05-30 15:04:34.665689299 +0800 CST m=+5.148623038 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:06:34.673 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:04:34.675 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:54:34.675 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214249080017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-0vnff, pid:9479, start at 2024-05-30 15:04:34.665689299 +0800 CST m=+5.148623038 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:06:34.673 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:04:34.675 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:54:34.675 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214249b00013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-0vnff, pid:9547, start at 2024-05-30 15:04:34.697674916 +0800 CST m=+5.131099941 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:06:34.704 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:04:34.668 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:54:34.668 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/owner_remove_table_error/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/owner_remove_table_error/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/owner_remove_table_error/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/owner_remove_table_error/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/owner_remove_table_error/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/autorandom/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table mark.finish_mark not exists for 2-th check, retry later table mark.finish_mark_3 not exists for 35-th check, retry later table consistent_replicate_storage_file_large_value.t2 exists check diff successfully wait process cdc.test exit for 1-th time... [Thu May 30 15:04:37 CST 2024] <<<<<< START cdc server in owner_remove_table_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/OwnerRemoveTableError=1*return(true)' + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_remove_table_error.1078110783.out server --log-file /tmp/tidb_cdc_test/owner_remove_table_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/owner_remove_table_error/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table mark.finish_mark not exists for 3-th check, retry later wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... table mark.finish_mark_3 not exists for 36-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:04:39 CST 2024] <<<<<< START cdc server in consistent_replicate_storage_file_large_value case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=return(true)' + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_storage_file_large_value.1445414456.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table mark.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... table mark.finish_mark_3 not exists for 37-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:04:40 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/950c4304-2481-4e01-bbc2-fc41890f88d4 {"id":"950c4304-2481-4e01-bbc2-fc41890f88d4","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8506872b7 950c4304-2481-4e01-bbc2-fc41890f88d4 /tidb/cdc/default/default/upstream/7374685046194215923 {"id":7374685046194215923,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/950c4304-2481-4e01-bbc2-fc41890f88d4 {"id":"950c4304-2481-4e01-bbc2-fc41890f88d4","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8506872b7 950c4304-2481-4e01-bbc2-fc41890f88d4 /tidb/cdc/default/default/upstream/7374685046194215923 {"id":7374685046194215923,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/950c4304-2481-4e01-bbc2-fc41890f88d4 {"id":"950c4304-2481-4e01-bbc2-fc41890f88d4","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8506872b7 950c4304-2481-4e01-bbc2-fc41890f88d4 /tidb/cdc/default/default/upstream/7374685046194215923 {"id":7374685046194215923,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table owner_remove_table_error.finished_mark not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:04:42 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/d6f0585b-08b4-469d-b0f1-cbf57f707955 {UpstreamID:7374684882919469615 Namespace:default ID:d6f0585b-08b4-469d-b0f1-cbf57f707955 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 15:04:06.616095315 +0800 CST StartTs:450115048952037385 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003dea240 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115048991621122} {CheckpointTs:450115053120389153 MinTableBarrierTs:450115058100862981 AdminJobType:noop} span: {table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}, resolvedTs: 450115058100862982, checkpointTs: 450115053120389153, state: Replicating span: {table_id:111,start_key:7480000000000000ff6f5f720000000000fa,end_key:7480000000000000ff6f5f730000000000fa}, resolvedTs: 450115058100862980, checkpointTs: 450115053120389153, state: Replicating span: {table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}, resolvedTs: 450115058100862982, checkpointTs: 450115053120389153, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115058100862982, checkpointTs: 450115053120389153, state: Replicating span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 450115058100862982, checkpointTs: 450115053120389153, state: Replicating span: {table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}, resolvedTs: 450115058100862982, checkpointTs: 450115053120389153, state: Replicating span: {table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}, resolvedTs: 450115058100862982, checkpointTs: 450115053120389153, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450115058100862982, checkpointTs: 450115053120389153, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b73739c0-9a0c-420e-afb4-e0d1c0ff1a63 {"id":"b73739c0-9a0c-420e-afb4-e0d1c0ff1a63","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84fd3ed25 b73739c0-9a0c-420e-afb4-e0d1c0ff1a63 /tidb/cdc/default/default/changefeed/info/d6f0585b-08b4-469d-b0f1-cbf57f707955 {"upstream-id":7374684882919469615,"namespace":"default","changefeed-id":"d6f0585b-08b4-469d-b0f1-cbf57f707955","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T15:04:06.616095315+08:00","start-ts":450115048952037385,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115048991621122} /tidb/cdc/default/default/changefeed/status/d6f0585b-08b4-469d-b0f1-cbf57f707955 {"checkpoint-ts":450115053120389153,"min-table-barrier-ts":450115058100862981,"admin-job-type":0} /tidb/cdc/default/default/task/position/b73739c0-9a0c-420e-afb4-e0d1c0ff1a63/d6f0585b-08b4-469d-b0f1-cbf57f707955 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684882919469615 {"id":7374684882919469615,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/d6f0585b-08b4-469d-b0f1-cbf57f707955 {UpstreamID:7374684882919469615 Namespace:default ID:d6f0585b-08b4-469d-b0f1-cbf57f707955 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 15:04:06.616095315 +0800 CST StartTs:450115048952037385 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003dea240 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115048991621122} {CheckpointTs:450115053120389153 MinTableBarrierTs:450115058100862981 AdminJobType:noop} span: {table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}, resolvedTs: 450115058100862982, checkpointTs: 450115053120389153, state: Replicating span: {table_id:111,start_key:7480000000000000ff6f5f720000000000fa,end_key:7480000000000000ff6f5f730000000000fa}, resolvedTs: 450115058100862980, checkpointTs: 450115053120389153, state: Replicating span: {table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}, resolvedTs: 450115058100862982, checkpointTs: 450115053120389153, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115058100862982, checkpointTs: 450115053120389153, state: Replicating span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 450115058100862982, checkpointTs: 450115053120389153, state: Replicating span: {table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}, resolvedTs: 450115058100862982, checkpointTs: 450115053120389153, state: Replicating span: {table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}, resolvedTs: 450115058100862982, checkpointTs: 450115053120389153, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450115058100862982, checkpointTs: 450115053120389153, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b73739c0-9a0c-420e-afb4-e0d1c0ff1a63 {"id":"b73739c0-9a0c-420e-afb4-e0d1c0ff1a63","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84fd3ed25 b73739c0-9a0c-420e-afb4-e0d1c0ff1a63 /tidb/cdc/default/default/changefeed/info/d6f0585b-08b4-469d-b0f1-cbf57f707955 {"upstream-id":7374684882919469615,"namespace":"default","changefeed-id":"d6f0585b-08b4-469d-b0f1-cbf57f707955","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T15:04:06.616095315+08:00","start-ts":450115048952037385,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115048991621122} + grep -q 'failed to get info:' /tidb/cdc/default/default/changefeed/status/d6f0585b-08b4-469d-b0f1-cbf57f707955 {"checkpoint-ts":450115053120389153,"min-table-barrier-ts":450115058100862981,"admin-job-type":0} /tidb/cdc/default/default/task/position/b73739c0-9a0c-420e-afb4-e0d1c0ff1a63/d6f0585b-08b4-469d-b0f1-cbf57f707955 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684882919469615 {"id":7374684882919469615,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/d6f0585b-08b4-469d-b0f1-cbf57f707955 {UpstreamID:7374684882919469615 Namespace:default ID:d6f0585b-08b4-469d-b0f1-cbf57f707955 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 15:04:06.616095315 +0800 CST StartTs:450115048952037385 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003dea240 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115048991621122} {CheckpointTs:450115053120389153 MinTableBarrierTs:450115058100862981 AdminJobType:noop} span: {table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}, resolvedTs: 450115058100862982, checkpointTs: 450115053120389153, state: Replicating span: {table_id:111,start_key:7480000000000000ff6f5f720000000000fa,end_key:7480000000000000ff6f5f730000000000fa}, resolvedTs: 450115058100862980, checkpointTs: 450115053120389153, state: Replicating span: {table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}, resolvedTs: 450115058100862982, checkpointTs: 450115053120389153, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115058100862982, checkpointTs: 450115053120389153, state: Replicating span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 450115058100862982, checkpointTs: 450115053120389153, state: Replicating span: {table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}, resolvedTs: 450115058100862982, checkpointTs: 450115053120389153, state: Replicating span: {table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}, resolvedTs: 450115058100862982, checkpointTs: 450115053120389153, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450115058100862982, checkpointTs: 450115053120389153, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b73739c0-9a0c-420e-afb4-e0d1c0ff1a63 {"id":"b73739c0-9a0c-420e-afb4-e0d1c0ff1a63","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84fd3ed25 b73739c0-9a0c-420e-afb4-e0d1c0ff1a63 /tidb/cdc/default/default/changefeed/info/d6f0585b-08b4-469d-b0f1-cbf57f707955 {"upstream-id":7374684882919469615,"namespace":"default","changefeed-id":"d6f0585b-08b4-469d-b0f1-cbf57f707955","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T15:04:06.616095315+08:00","start-ts":450115048952037385,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115048991621122} /tidb/cdc/default/default/changefeed/status/d6f0585b-08b4-469d-b0f1-cbf57f707955 {"checkpoint-ts":450115053120389153,"min-table-barrier-ts":450115058100862981,"admin-job-type":0} /tidb/cdc/default/default/task/position/b73739c0-9a0c-420e-afb4-e0d1c0ff1a63/d6f0585b-08b4-469d-b0f1-cbf57f707955 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684882919469615 {"id":7374684882919469615,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.cli.21869.out cli tso query --pd=http://127.0.0.1:2379 cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:04:42 CST 2024] <<<<<< run test case multi_source success! >>>>>> table mark.finish_mark_3 not exists for 38-th check, retry later table owner_remove_table_error.finished_mark not exists for 2-th check, retry later + set +x + tso='450115058449514502 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115058449514502 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x table mark.finish_mark_3 not exists for 39-th check, retry later [Thu May 30 15:04:45 CST 2024] <<<<<< START cdc server in split_region case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.2190421906.out server --log-file /tmp/tidb_cdc_test/split_region/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/split_region/cdc_data --cluster-id default + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table owner_remove_table_error.finished_mark not exists for 3-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/autorandom Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... table mark.finish_mark_3 not exists for 40-th check, retry later table owner_remove_table_error.finished_mark exists check diff successfully Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... table mark.finish_mark_3 not exists for 41-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:04:48 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/51a1eca8-7cbd-4562-b35a-e066eaa361f6 {"id":"51a1eca8-7cbd-4562-b35a-e066eaa361f6","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc85046e013 51a1eca8-7cbd-4562-b35a-e066eaa361f6 /tidb/cdc/default/default/upstream/7374685011277253817 {"id":7374685011277253817,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/51a1eca8-7cbd-4562-b35a-e066eaa361f6 {"id":"51a1eca8-7cbd-4562-b35a-e066eaa361f6","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc85046e013 51a1eca8-7cbd-4562-b35a-e066eaa361f6 /tidb/cdc/default/default/upstream/7374685011277253817 {"id":7374685011277253817,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/51a1eca8-7cbd-4562-b35a-e066eaa361f6 {"id":"51a1eca8-7cbd-4562-b35a-e066eaa361f6","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc85046e013 51a1eca8-7cbd-4562-b35a-e066eaa361f6 /tidb/cdc/default/default/upstream/7374685011277253817 {"id":7374685011277253817,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.cli.21951.out cli changefeed create --start-ts=450115058449514502 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 15:04:49 CST 2024] <<<<<< run test case owner_remove_table_error success! >>>>>> Create changefeed successfully! ID: 37132804-3acb-45e4-a641-81e70fa35c86 Info: {"upstream_id":7374685011277253817,"namespace":"default","id":"37132804-3acb-45e4-a641-81e70fa35c86","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T15:04:49.42624371+08:00","start_ts":450115058449514502,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115058449514502,"checkpoint_ts":450115058449514502,"checkpoint_time":"2024-05-30 15:04:42.684"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x table split_region.test1 not exists for 1-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_3 not exists for 42-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table split_region.test1 exists table split_region.test2 not exists for 1-th check, retry later table mark.finish_mark_3 not exists for 43-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_3 not exists for 44-th check, retry later table split_region.test2 exists check diff successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_3 not exists for 45-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214397d80010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t2zx2, pid:26382, start at 2024-05-30 15:04:56.065490491 +0800 CST m=+5.514173833 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:06:56.075 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:04:56.054 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:54:56.054 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214397d80010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t2zx2, pid:26382, start at 2024-05-30 15:04:56.065490491 +0800 CST m=+5.514173833 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:06:56.075 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:04:56.054 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:54:56.054 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214397c00011 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t2zx2, pid:26453, start at 2024-05-30 15:04:56.066990647 +0800 CST m=+5.430691197 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:06:56.074 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:04:56.048 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:54:56.048 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/autorandom/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/autorandom/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table mark.finish_mark_3 not exists for 46-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Thu May 30 15:05:00 CST 2024] <<<<<< START cdc server in autorandom case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.autorandom.2782027822.out server --log-file /tmp/tidb_cdc_test/autorandom/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/autorandom/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table mark.finish_mark_3 not exists for 47-th check, retry later table mark.finish_mark_3 not exists for 48-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:05:04 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a944111c-cc43-4789-baf9-5e5934c67328 {"id":"a944111c-cc43-4789-baf9-5e5934c67328","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc850c06eb8 a944111c-cc43-4789-baf9-5e5934c67328 /tidb/cdc/default/default/upstream/7374685137994644533 {"id":7374685137994644533,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a944111c-cc43-4789-baf9-5e5934c67328 {"id":"a944111c-cc43-4789-baf9-5e5934c67328","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc850c06eb8 a944111c-cc43-4789-baf9-5e5934c67328 /tidb/cdc/default/default/upstream/7374685137994644533 {"id":7374685137994644533,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a944111c-cc43-4789-baf9-5e5934c67328 {"id":"a944111c-cc43-4789-baf9-5e5934c67328","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc850c06eb8 a944111c-cc43-4789-baf9-5e5934c67328 /tidb/cdc/default/default/upstream/7374685137994644533 {"id":7374685137994644533,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: d10ab3d3-01fa-4b5b-ac3c-68a5b4a113ac Info: {"upstream_id":7374685137994644533,"namespace":"default","id":"d10ab3d3-01fa-4b5b-ac3c-68a5b4a113ac","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T15:05:04.960780727+08:00","start_ts":450115064261246979,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115064261246979,"checkpoint_ts":450115064261246979,"checkpoint_time":"2024-05-30 15:05:04.854"} table mark.finish_mark_3 not exists for 49-th check, retry later table autorandom_test.table_a not exists for 1-th check, retry later table mark.finish_mark_3 not exists for 50-th check, retry later table autorandom_test.table_a exists check diff failed 1-th time, retry later table mark.finish_mark_3 not exists for 51-th check, retry later check_redo_resolved_ts d6f0585b-08b4-469d-b0f1-cbf57f707955 450115064772427778 file:///tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/redo /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc_data/redo/d6f0585b-08b4-469d-b0f1-cbf57f707955/meta global resolved ts 450115064405426178 not forward to 450115064772427778 run task failed 1-th time, retry later check_redo_resolved_ts d6f0585b-08b4-469d-b0f1-cbf57f707955 450115064772427778 file:///tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/redo /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc_data/redo/d6f0585b-08b4-469d-b0f1-cbf57f707955/meta run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... check diff successfully wait process cdc.test exit for 3-th time... wait process cdc.test exit for 1-th time... check diff failed 1-th time, retry later check diff successfully cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [2024/05/30 15:05:11.119 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2024/05/30 15:05:11.120 +08:00] [INFO] [apply.go:78] ["set memory limit"] [memoryLimit=10737418240] [2024/05/30 15:05:11.120 +08:00] [INFO] [apply.go:94] ["Start http pprof server"] [addr=:6060] [2024/05/30 15:05:11.120 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2024/05/30 15:05:11.120 +08:00] [INFO] [redo.go:167] ["apply redo log starts"] [checkpointTs=450115059280510979] [resolvedTs=450115064929976322] [2024/05/30 15:05:11.121 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=704.947ยตs] [2024/05/30 15:05:11.121 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=729.499ยตs] [2024/05/30 15:05:11.122 +08:00] [INFO] [tz.go:36] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/05/30 15:05:11.122 +08:00] [WARN] [config.go:394] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/05/30 15:05:11.122 +08:00] [ERROR] [config.go:209] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2024/05/30 15:05:11.126 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22NO_ENGINE_SUBSTITUTION%2CNO_AUTO_VALUE_ON_ZERO%2CIGNORE_SPACE%2CONLY_FULL_GROUP_BY%2CALLOW_INVALID_DATES%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/05/30 15:05:11.128 +08:00] [INFO] [mysql.go:198] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2024/05/30 15:05:11.129 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:05:11.129 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:05:11.129 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:05:11.129 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:05:11.129 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:05:11.129 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:05:11.129 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:05:11.129 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:05:11.129 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:05:11.129 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:05:11.129 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:05:11.130 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:05:11.130 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:05:11.130 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:05:11.130 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:05:11.130 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:05:11.130 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2024/05/30 15:05:11.130 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2024/05/30 15:05:11.130 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2024/05/30 15:05:11.130 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2024/05/30 15:05:11.130 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2024/05/30 15:05:11.130 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2024/05/30 15:05:11.130 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2024/05/30 15:05:11.130 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2024/05/30 15:05:11.130 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2024/05/30 15:05:11.130 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2024/05/30 15:05:11.130 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2024/05/30 15:05:11.130 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2024/05/30 15:05:11.130 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2024/05/30 15:05:11.130 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2024/05/30 15:05:11.130 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2024/05/30 15:05:11.130 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2024/05/30 15:05:11.130 +08:00] [INFO] [tz.go:36] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/05/30 15:05:11.130 +08:00] [WARN] [config.go:394] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/05/30 15:05:11.130 +08:00] [ERROR] [config.go:209] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2024/05/30 15:05:11.133 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22NO_ENGINE_SUBSTITUTION%2CNO_AUTO_VALUE_ON_ZERO%2CIGNORE_SPACE%2CONLY_FULL_GROUP_BY%2CALLOW_INVALID_DATES%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/05/30 15:05:11.135 +08:00] [INFO] [mysql_ddl_sink.go:99] ["MySQL DDL sink is created"] [namespace=] [changefeed=] [2024/05/30 15:05:11.135 +08:00] [ERROR] [redo.go:273] ["ignore unsupported DDL"] [ddl="{\"StartTs\":450115059241189388,\"CommitTs\":450115059280510979,\"Query\":\"ALTER TABLE `consistent_replicate_storage_file_large_value`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `consistent_replicate_storage_file_large_value`.`t2`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_storage_file_large_value\",\"tbl-name\":\"t1\",\"tbl-id\":108,\"is-partition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":42,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [2024/05/30 15:05:11.136 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=295] [newQuota=75520] table mark.finish_mark_3 not exists for 52-th check, retry later wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... [2024/05/30 15:05:11.336 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}] [checkpointTs=450115064929976322] [2024/05/30 15:05:11.336 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}] [checkpointTs=450115064929976322] [2024/05/30 15:05:11.637 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}] [checkpointTs=450115064929976322] [2024/05/30 15:05:11.637 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}] [checkpointTs=450115064929976322] [2024/05/30 15:05:11.737 +08:00] [INFO] [redo.go:223] ["apply redo log finishes"] [appliedLogCount=8] [appliedDDLCount=0] [currentCheckpoint=450115064929976322] [2024/05/30 15:05:11.737 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2024/05/30 15:05:11.737 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2024/05/30 15:05:11.737 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2024/05/30 15:05:11.737 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2024/05/30 15:05:11.737 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2024/05/30 15:05:11.737 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2024/05/30 15:05:11.737 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2024/05/30 15:05:11.737 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2024/05/30 15:05:11.737 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] [2024/05/30 15:05:11.737 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2024/05/30 15:05:11.737 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2024/05/30 15:05:11.737 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2024/05/30 15:05:11.737 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] [2024/05/30 15:05:11.737 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] [2024/05/30 15:05:11.737 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2024/05/30 15:05:11.737 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] Apply redo log successfully wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... check diff successfully [Thu May 30 15:05:12 CST 2024] <<<<<< run test case consistent_replicate_storage_file_large_value success! >>>>>> count(*) 0 wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:05:12 CST 2024] <<<<<< run test case autorandom success! >>>>>> cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:05:12 CST 2024] <<<<<< run test case split_region success! >>>>>> table mark.finish_mark_3 not exists for 53-th check, retry later table mark.finish_mark_3 not exists for 54-th check, retry later table mark.finish_mark_3 not exists for 55-th check, retry later table mark.finish_mark_3 not exists for 56-th check, retry later table mark.finish_mark_3 not exists for 57-th check, retry later table mark.finish_mark_3 not exists for 58-th check, retry later table mark.finish_mark_3 not exists for 59-th check, retry later table mark.finish_mark_3 not exists for 60-th check, retry later table mark.finish_mark_3 not exists for 61-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/consistent_replicate_storage_s3/run.sh using Sink-Type: mysql... <<================= * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:24927; Connection refused * Closing connection 0 You are running an older version of MinIO released 3 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:24927 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 17D4325C2D2C76A0 < X-Xss-Protection: 1; mode=block < Date: Thu, 30 May 2024 07:05:29 GMT < { [data not shown] * Connection #0 to host 127.0.0.1 left intact Bucket 's3://logbucket/' created The 1 times to try to start tidb cluster... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/generate_column/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } table mark.finish_mark_3 not exists for 62-th check, retry later [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } start tidb cluster in /tmp/tidb_cdc_test/consistent_replicate_storage_s3 Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... table mark.finish_mark_3 exists table mark.finish_mark not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/generate_column Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... table mark.finish_mark exists check diff successfully Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release wait process cdc.test exit for 1-th time... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... wait process cdc.test exit for 4-th time... cdc.test: no process found wait process cdc.test exit for 5-th time... process cdc.test already exit [Thu May 30 15:05:38 CST 2024] <<<<<< run test case default_value success! >>>>>> ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2635 0 --:--:-- --:--:-- --:--:-- 2662 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-30 15:05:26.605","puller_resolved_ts":"2024-05-30 15:05:18.655","last_synced_ts":"2024-05-30 15:03:09.754","now_ts":"2024-05-30 15:05:26.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:05:26.605","puller_resolved_ts":"2024-05-30' '15:05:18.655","last_synced_ts":"2024-05-30' '15:03:09.754","now_ts":"2024-05-30' '15:05:26.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' + kill_pd ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 14403 7.8 0.0 13493256 142652 ? Sl 15:02 0:12 pd-server --advertise-client-urls http://127.0.0.1:2379 --client-urls http://0.0.0.0:2379 --advertise-peer-urls http://127.0.0.1:2380 --peer-urls http://0.0.0.0:2380 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/pd1.log --data-dir /tmp/tidb_cdc_test/synced_status/pd1 --name=pd1 --initial-cluster=pd1=http://127.0.0.1:2380 jenkins 14448 5.5 0.0 13902480 133600 ? Sl 15:02 0:08 pd-server --advertise-client-urls http://127.0.0.1:2479 --client-urls http://0.0.0.0:2479 --advertise-peer-urls http://127.0.0.1:2480 --peer-urls http://0.0.0.0:2480 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/down_pd.log --data-dir /tmp/tidb_cdc_test/synced_status/down_pd' ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 + sleep 20 {"level":"warn","ts":1717052734.464516,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00242c1c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1717052734.4646106,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1717052734.644903,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0024b0700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1717052734.6449864,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1717052734.798321,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00299c380/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1717052734.798394,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore {"level":"warn","ts":"2024-05-30T15:05:39.501585+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000543180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:05:39.504953+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f60000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) {"level":"warn","ts":"2024-05-30T15:05:39.605031+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f96a80/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check_changefeed_state http://127.0.0.1:2379 6e72ccbb-4fd3-4f2f-a139-35d70277c304 finished null + endpoints=http://127.0.0.1:2379 + changefeed_id=6e72ccbb-4fd3-4f2f-a139-35d70277c304 + expected_state=finished + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 6e72ccbb-4fd3-4f2f-a139-35d70277c304 -s + info='{ "upstream_id": 7374684867512116127, "namespace": "default", "id": "6e72ccbb-4fd3-4f2f-a139-35d70277c304", "state": "finished", "checkpoint_tso": 450115070659919874, "checkpoint_time": "2024-05-30 15:05:29.263", "error": null }' + echo '{ "upstream_id": 7374684867512116127, "namespace": "default", "id": "6e72ccbb-4fd3-4f2f-a139-35d70277c304", "state": "finished", "checkpoint_tso": 450115070659919874, "checkpoint_time": "2024-05-30 15:05:29.263", "error": null }' { "upstream_id": 7374684867512116127, "namespace": "default", "id": "6e72ccbb-4fd3-4f2f-a139-35d70277c304", "state": "finished", "checkpoint_tso": 450115070659919874, "checkpoint_time": "2024-05-30 15:05:29.263", "error": null } ++ echo '{' '"upstream_id":' 7374684867512116127, '"namespace":' '"default",' '"id":' '"6e72ccbb-4fd3-4f2f-a139-35d70277c304",' '"state":' '"finished",' '"checkpoint_tso":' 450115070659919874, '"checkpoint_time":' '"2024-05-30' '15:05:29.263",' '"error":' null '}' ++ jq -r .state + state=finished + [[ ! finished == \f\i\n\i\s\h\e\d ]] ++ echo '{' '"upstream_id":' 7374684867512116127, '"namespace":' '"default",' '"id":' '"6e72ccbb-4fd3-4f2f-a139-35d70277c304",' '"state":' '"finished",' '"checkpoint_tso":' 450115070659919874, '"checkpoint_time":' '"2024-05-30' '15:05:29.263",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:05:37 CST 2024] <<<<<< run test case changefeed_finish success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2146877c0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t2zx2, pid:28922, start at 2024-05-30 15:05:44.207341237 +0800 CST m=+5.346085896 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:07:44.215 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:05:44.208 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:55:44.208 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2146877c0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t2zx2, pid:28922, start at 2024-05-30 15:05:44.207341237 +0800 CST m=+5.346085896 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:07:44.215 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:05:44.208 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:55:44.208 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214688f8000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-t2zx2, pid:29008, start at 2024-05-30 15:05:44.268565372 +0800 CST m=+5.319486388 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:07:44.276 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:05:44.254 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:55:44.254 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/generate_column/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/generate_column/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/generate_column/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/generate_column/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/generate_column/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21468f140013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3, pid:15669, start at 2024-05-30 15:05:44.66918641 +0800 CST m=+5.272518669 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:07:44.677 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:05:44.645 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:55:44.645 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21468f140013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3, pid:15669, start at 2024-05-30 15:05:44.66918641 +0800 CST m=+5.272518669 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:07:44.677 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:05:44.645 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:55:44.645 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2146907c0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3, pid:15737, start at 2024-05-30 15:05:44.762162768 +0800 CST m=+5.308602379 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:07:44.769 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:05:44.735 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:55:44.735 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_replicate_storage_s3/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_replicate_storage_s3/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_s3/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_s3/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_s3/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } {"level":"warn","ts":"2024-05-30T15:05:45.502347+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000543180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:05:45.505731+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f60000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:05:45.605709+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f96a80/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/force_replicate_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.generate_column.cli.30263.out cli tso query --pd=http://127.0.0.1:2379 [Thu May 30 15:05:47 CST 2024] <<<<<< START cdc server in consistent_replicate_storage_s3 case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_storage_s3.1711117113.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/simple/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + set +x + tso='450115075534749697 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115075534749697 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 15:05:49 CST 2024] <<<<<< START cdc server in generate_column case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.generate_column.3030230304.out server --log-file /tmp/tidb_cdc_test/generate_column/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/generate_column/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + grep -q 'failed to get info:' + echo '' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 start tidb cluster in /tmp/tidb_cdc_test/force_replicate_table Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:05:50 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cf79692b-a2c5-4a3a-bfd3-1a19c595a366 {"id":"cf79692b-a2c5-4a3a-bfd3-1a19c595a366","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc851763eb6 cf79692b-a2c5-4a3a-bfd3-1a19c595a366 /tidb/cdc/default/default/upstream/7374685343960551859 {"id":7374685343960551859,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cf79692b-a2c5-4a3a-bfd3-1a19c595a366 {"id":"cf79692b-a2c5-4a3a-bfd3-1a19c595a366","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc851763eb6 cf79692b-a2c5-4a3a-bfd3-1a19c595a366 /tidb/cdc/default/default/upstream/7374685343960551859 {"id":7374685343960551859,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cf79692b-a2c5-4a3a-bfd3-1a19c595a366 {"id":"cf79692b-a2c5-4a3a-bfd3-1a19c595a366","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc851763eb6 cf79692b-a2c5-4a3a-bfd3-1a19c595a366 /tidb/cdc/default/default/upstream/7374685343960551859 {"id":7374685343960551859,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x start tidb cluster in /tmp/tidb_cdc_test/simple Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:05:51.503618+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000543180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:05:51.506781+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f60000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:05:51.607145+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f96a80/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ***************** properties ***************** "mysql.db"="consistent_replicate_storage_s3" "workload"="core" "readallfields"="true" "readproportion"="0" "operationcount"="0" "updateproportion"="0" "mysql.port"="4000" "requestdistribution"="uniform" "dotransactions"="false" "scanproportion"="0" "mysql.user"="root" "recordcount"="5000" "mysql.host"="127.0.0.1" "threadcount"="10" "fieldcount"="100" "insertproportion"="0" ********************************************** + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:05:52 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fa75317b-7ce5-4db4-8158-cb6fc0518331 {"id":"fa75317b-7ce5-4db4-8158-cb6fc0518331","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8517745be fa75317b-7ce5-4db4-8158-cb6fc0518331 /tidb/cdc/default/default/upstream/7374685344961014683 {"id":7374685344961014683,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fa75317b-7ce5-4db4-8158-cb6fc0518331 {"id":"fa75317b-7ce5-4db4-8158-cb6fc0518331","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8517745be fa75317b-7ce5-4db4-8158-cb6fc0518331 /tidb/cdc/default/default/upstream/7374685344961014683 {"id":7374685344961014683,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fa75317b-7ce5-4db4-8158-cb6fc0518331 {"id":"fa75317b-7ce5-4db4-8158-cb6fc0518331","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8517745be fa75317b-7ce5-4db4-8158-cb6fc0518331 /tidb/cdc/default/default/upstream/7374685344961014683 {"id":7374685344961014683,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.generate_column.cli.30351.out cli changefeed create --start-ts=450115075534749697 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ Verifying downstream PD is started... Create changefeed successfully! ID: 9c542e8d-3cea-4afb-b876-59cfe1dbbca3 Info: {"upstream_id":7374685344961014683,"namespace":"default","id":"9c542e8d-3cea-4afb-b876-59cfe1dbbca3","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T15:05:52.812251805+08:00","start_ts":450115075534749697,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115075534749697,"checkpoint_ts":450115075534749697,"checkpoint_time":"2024-05-30 15:05:47.859"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Run finished, takes 2.154270155s INSERT - Takes(s): 2.1, Count: 5000, OPS: 2336.4, Avg(us): 3909, Min(us): 1426, Max(us): 59789, 95th(us): 5000, 99th(us): 50000 table generate_column.t not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_storage_s3.usertable exists table consistent_replicate_storage_s3.check1 not exists for 1-th check, retry later table generate_column.t exists table generate_column.t1 not exists for 1-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:05:57.505385+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000543180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:05:57.50734+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f60000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:05:57.608821+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f96a80/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_storage_s3.check1 not exists for 2-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:05:59.491969+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f60000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2024-05-30T15:05:59.492032+0800","logger":"etcd-client","caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-05-30T15:05:59.493356+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000543180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2024-05-30T15:05:59.493395+0800","logger":"etcd-client","caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-05-30T15:05:59.594503+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f96a80/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":"2024-05-30T15:05:59.594562+0800","logger":"etcd-client","caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} table generate_column.t1 exists check diff successfully wait process cdc.test exit for 1-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_storage_s3.check1 not exists for 3-th check, retry later wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:06:01 CST 2024] <<<<<< run test case generate_column success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214774340012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-nk3lw, pid:6709, start at 2024-05-30 15:05:59.334161047 +0800 CST m=+5.472898357 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:07:59.342 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:05:59.309 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:55:59.309 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214774340012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-nk3lw, pid:6709, start at 2024-05-30 15:05:59.334161047 +0800 CST m=+5.472898357 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:07:59.342 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:05:59.309 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:55:59.309 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214774280013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-nk3lw, pid:6777, start at 2024-05-30 15:05:59.334766037 +0800 CST m=+5.386467656 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:07:59.342 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:05:59.306 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:55:59.306 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/force_replicate_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/force_replicate_table/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_storage_s3.check1 not exists for 4-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:14 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:15 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:06:03.506184+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000543180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:06:03.509018+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f60000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:06:03.610501+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f96a80/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} [Thu May 30 15:06:04 CST 2024] <<<<<< START cdc server in force_replicate_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.force_replicate_table.80528054.out server --log-file /tmp/tidb_cdc_test/force_replicate_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/force_replicate_table/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214798dc0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l2n7g, pid:20235, start at 2024-05-30 15:06:01.68623657 +0800 CST m=+5.300816739 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:08:01.694 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:06:01.655 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:56:01.655 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214798dc0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l2n7g, pid:20235, start at 2024-05-30 15:06:01.68623657 +0800 CST m=+5.300816739 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:08:01.694 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:06:01.655 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:56:01.655 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21479a58000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l2n7g, pid:20319, start at 2024-05-30 15:06:01.761391896 +0800 CST m=+5.313558865 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:08:01.768 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:06:01.750 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:56:01.750 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/simple/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/simple/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/simple/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/simple/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/simple/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table consistent_replicate_storage_s3.check1 not exists for 5-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.cli.21569.out cli tso query --pd=http://127.0.0.1:2379 table consistent_replicate_storage_s3.check1 not exists for 6-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:06:07 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6b4e5c64-d6bc-44a1-94c3-e2093c45400f {"id":"6b4e5c64-d6bc-44a1-94c3-e2093c45400f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc851b7b7be 6b4e5c64-d6bc-44a1-94c3-e2093c45400f /tidb/cdc/default/default/upstream/7374685408348016378 {"id":7374685408348016378,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6b4e5c64-d6bc-44a1-94c3-e2093c45400f {"id":"6b4e5c64-d6bc-44a1-94c3-e2093c45400f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc851b7b7be 6b4e5c64-d6bc-44a1-94c3-e2093c45400f /tidb/cdc/default/default/upstream/7374685408348016378 {"id":7374685408348016378,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6b4e5c64-d6bc-44a1-94c3-e2093c45400f {"id":"6b4e5c64-d6bc-44a1-94c3-e2093c45400f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc851b7b7be 6b4e5c64-d6bc-44a1-94c3-e2093c45400f /tidb/cdc/default/default/upstream/7374685408348016378 + grep -q 'etcd info' {"id":7374685408348016378,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x Create changefeed successfully! ID: 1f2d53b7-92f6-4a96-b3db-415da3a8bdc1 Info: {"upstream_id":7374685408348016378,"namespace":"default","id":"1f2d53b7-92f6-4a96-b3db-415da3a8bdc1","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?safe-mode=true","create_time":"2024-05-30T15:06:07.650857448+08:00","start_ts":450115079847018504,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":true,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115079847018504,"checkpoint_ts":450115079847018504,"checkpoint_time":"2024-05-30 15:06:04.309"} + set +x + tso='450115080566865921 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115080566865921 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x table consistent_replicate_storage_s3.check1 not exists for 7-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:17 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:18 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:21 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:22 --:--:-- 0{"level":"warn","ts":1717052769.4655766,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00242c1c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1717052769.4656308,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-05-30T15:06:09.507183+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000543180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} [Thu May 30 15:06:09 CST 2024] <<<<<< START cdc server in simple case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.2160821610.out server --log-file /tmp/tidb_cdc_test/simple/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/simple/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 {"level":"warn","ts":"2024-05-30T15:06:09.510536+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f60000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:06:09.610716+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f96a80/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"warn","ts":1717052769.6460567,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0024b0700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1717052769.6461072,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1717052769.7994258,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00299c380/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1717052769.7994862,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} table consistent_replicate_storage_s3.check1 not exists for 8-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:06:12 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0cd74a4c-a864-4fc5-96c1-10c7654e668c {"id":"0cd74a4c-a864-4fc5-96c1-10c7654e668c","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc851bd90e5 0cd74a4c-a864-4fc5-96c1-10c7654e668c /tidb/cdc/default/default/upstream/7374685414111196882 {"id":7374685414111196882,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0cd74a4c-a864-4fc5-96c1-10c7654e668c {"id":"0cd74a4c-a864-4fc5-96c1-10c7654e668c","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc851bd90e5 0cd74a4c-a864-4fc5-96c1-10c7654e668c /tidb/cdc/default/default/upstream/7374685414111196882 {"id":7374685414111196882,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0cd74a4c-a864-4fc5-96c1-10c7654e668c {"id":"0cd74a4c-a864-4fc5-96c1-10c7654e668c","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc851bd90e5 0cd74a4c-a864-4fc5-96c1-10c7654e668c /tidb/cdc/default/default/upstream/7374685414111196882 {"id":7374685414111196882,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.cli.21654.out cli changefeed create --start-ts=450115080566865921 --sink-uri=mysql+ssl://normal:123456@127.0.0.1:3306/ table consistent_replicate_storage_s3.check1 not exists for 9-th check, retry later Create changefeed successfully! ID: ac125328-10b1-425e-88b1-a80cdbca810c Info: {"upstream_id":7374685414111196882,"namespace":"default","id":"ac125328-10b1-425e-88b1-a80cdbca810c","sink_uri":"mysql+ssl://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T15:06:13.117646661+08:00","start_ts":450115080566865921,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115080566865921,"checkpoint_ts":450115080566865921,"checkpoint_time":"2024-05-30 15:06:07.055"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x succeed to verify meta placement rules TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 1-th time, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:23 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:24 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:25 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:26 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:27 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:28 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:06:15.508813+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000543180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} table consistent_replicate_storage_s3.check1 not exists for 10-th check, retry later {"level":"warn","ts":"2024-05-30T15:06:15.511716+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f60000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} table force_replicate_table.t0 exists table force_replicate_table.t1 exists table force_replicate_table.t2 exists table force_replicate_table.t3 not exists for 1-th check, retry later {"level":"warn","ts":"2024-05-30T15:06:15.611888+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f96a80/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} table force_replicate_table.t3 exists table force_replicate_table.t4 not exists for 1-th check, retry later TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 2-th time, retry later table consistent_replicate_storage_s3.check1 not exists for 11-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 27 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 33 + synced_status='{ "error_msg": "[CDC:ErrPDEtcdAPIError]etcd api call error: context deadline exceeded", "error_code": "CDC:ErrPDEtcdAPIError" }' ++ echo '{' '"error_msg":' '"[CDC:ErrPDEtcdAPIError]etcd' api call error: context deadline 'exceeded",' '"error_code":' '"CDC:ErrPDEtcdAPIError"' '}' ++ jq -r .error_code + error_code=CDC:ErrPDEtcdAPIError + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster table force_replicate_table.t4 exists table force_replicate_table.t5 not exists for 1-th check, retry later check data successfully wait process cdc.test exit for 1-th time... table consistent_replicate_storage_s3.check1 not exists for 12-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } wait process cdc.test exit for 2-th time... [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 15:06:20 CST 2024] <<<<<< run test case simple success! >>>>>> table force_replicate_table.t5 exists table force_replicate_table.t6 not exists for 1-th check, retry later table consistent_replicate_storage_s3.check1 not exists for 13-th check, retry later table force_replicate_table.t6 exists check_data_subset force_replicate_table.t0 127.0.0.1 4000 127.0.0.1 3306 table consistent_replicate_storage_s3.check1 exists table consistent_replicate_storage_s3.t2 not exists for 1-th check, retry later run task successfully check_data_subset force_replicate_table.t1 127.0.0.1 4000 127.0.0.1 3306 run task successfully check_data_subset force_replicate_table.t2 127.0.0.1 4000 127.0.0.1 3306 table consistent_replicate_storage_s3.t2 not exists for 2-th check, retry later run task successfully check_data_subset force_replicate_table.t3 127.0.0.1 4000 127.0.0.1 3306 run task successfully check_data_subset force_replicate_table.t4 127.0.0.1 4000 127.0.0.1 3306 run task successfully check_data_subset force_replicate_table.t5 127.0.0.1 4000 127.0.0.1 3306 table consistent_replicate_storage_s3.t2 not exists for 3-th check, retry later run task successfully check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 id=19,a=NULL doesn't exist in downstream table force_replicate_table.t6 run task failed 1-th time, retry later + run_case_with_unavailable_tikv conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory table consistent_replicate_storage_s3.t2 not exists for 4-th check, retry later check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/cdc_server_tips/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... run task successfully wait process cdc.test exit for 1-th time... table consistent_replicate_storage_s3.t2 not exists for 5-th check, retry later wait process cdc.test exit for 2-th time... chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:06:32 CST 2024] <<<<<< run test case force_replicate_table success! >>>>>> start tidb cluster in /tmp/tidb_cdc_test/cdc_server_tips Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... table consistent_replicate_storage_s3.t2 not exists for 6-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table consistent_replicate_storage_s3.t2 exists check diff successfully wait process cdc.test exit for 1-th time... Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore wait process cdc.test exit for 2-th time... Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 3-th time... Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:06:37 CST 2024] <<<<<< START cdc server in consistent_replicate_storage_s3 case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_storage_s3.1746417466.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:06:42 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/9345ff97-0246-4c44-8259-0f8f82516588 {UpstreamID:7374685343960551859 Namespace:default ID:9345ff97-0246-4c44-8259-0f8f82516588 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 15:05:51.102268174 +0800 CST StartTs:450115076356833283 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004594090 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115076382785537} {CheckpointTs:450115089516462086 MinTableBarrierTs:450115089791713281 AdminJobType:noop} span: {table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}, resolvedTs: 450115089516462087, checkpointTs: 450115089516462086, state: Replicating span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 450115089516462087, checkpointTs: 450115089516462086, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115089516462087, checkpointTs: 450115089516462086, state: Replicating span: {table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}, resolvedTs: 450115089516462087, checkpointTs: 450115089516462086, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450115089516462087, checkpointTs: 450115089516462086, state: Replicating span: {table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}, resolvedTs: 450115089516462087, checkpointTs: 450115089516462086, state: Replicating span: {table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}, resolvedTs: 450115089516462087, checkpointTs: 450115089516462086, state: Replicating span: {table_id:111,start_key:7480000000000000ff6f5f720000000000fa,end_key:7480000000000000ff6f5f730000000000fa}, resolvedTs: 450115089516462087, checkpointTs: 450115089516462086, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/76c84694-c0ae-4b57-9564-49d8bb27547a {"id":"76c84694-c0ae-4b57-9564-49d8bb27547a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc85176405f 76c84694-c0ae-4b57-9564-49d8bb27547a /tidb/cdc/default/default/changefeed/info/9345ff97-0246-4c44-8259-0f8f82516588 {"upstream-id":7374685343960551859,"namespace":"default","changefeed-id":"9345ff97-0246-4c44-8259-0f8f82516588","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T15:05:51.102268174+08:00","start-ts":450115076356833283,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/","use-file-backend":false,"compression":"","flush-concurrency":2,"memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115076382785537} /tidb/cdc/default/default/changefeed/status/9345ff97-0246-4c44-8259-0f8f82516588 {"checkpoint-ts":450115089516462086,"min-table-barrier-ts":450115089791713281,"admin-job-type":0} /tidb/cdc/default/default/task/position/76c84694-c0ae-4b57-9564-49d8bb27547a/9345ff97-0246-4c44-8259-0f8f82516588 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374685343960551859 {"id":7374685343960551859,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/9345ff97-0246-4c44-8259-0f8f82516588 {UpstreamID:7374685343960551859 Namespace:default ID:9345ff97-0246-4c44-8259-0f8f82516588 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 15:05:51.102268174 +0800 CST StartTs:450115076356833283 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004594090 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115076382785537} {CheckpointTs:450115089516462086 MinTableBarrierTs:450115089791713281 AdminJobType:noop} span: {table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}, resolvedTs: 450115089516462087, checkpointTs: 450115089516462086, state: Replicating span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 450115089516462087, checkpointTs: 450115089516462086, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115089516462087, checkpointTs: 450115089516462086, state: Replicating span: {table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}, resolvedTs: 450115089516462087, checkpointTs: 450115089516462086, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450115089516462087, checkpointTs: 450115089516462086, state: Replicating span: {table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}, resolvedTs: 450115089516462087, checkpointTs: 450115089516462086, state: Replicating span: {table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}, resolvedTs: 450115089516462087, checkpointTs: 450115089516462086, state: Replicating span: {table_id:111,start_key:7480000000000000ff6f5f720000000000fa,end_key:7480000000000000ff6f5f730000000000fa}, resolvedTs: 450115089516462087, checkpointTs: 450115089516462086, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/76c84694-c0ae-4b57-9564-49d8bb27547a {"id":"76c84694-c0ae-4b57-9564-49d8bb27547a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc85176405f 76c84694-c0ae-4b57-9564-49d8bb27547a /tidb/cdc/default/default/changefeed/info/9345ff97-0246-4c44-8259-0f8f82516588 {"upstream-id":7374685343960551859,"namespace":"default","changefeed-id":"9345ff97-0246-4c44-8259-0f8f82516588","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T15:05:51.102268174+08:00","start-ts":450115076356833283,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/","use-file-backend":false,"compression":"","flush-concurrency":2,"memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115076382785537} /tidb/cdc/default/default/changefeed/status/9345ff97-0246-4c44-8259-0f8f82516588 {"checkpoint-ts":450115089516462086,"min-table-barrier-ts":450115089791713281,"admin-job-type":0} /tidb/cdc/default/default/task/position/76c84694-c0ae-4b57-9564-49d8bb27547a/9345ff97-0246-4c44-8259-0f8f82516588 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374685343960551859 {"id":7374685343960551859,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/9345ff97-0246-4c44-8259-0f8f82516588 {UpstreamID:7374685343960551859 Namespace:default ID:9345ff97-0246-4c44-8259-0f8f82516588 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 15:05:51.102268174 +0800 CST StartTs:450115076356833283 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004594090 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115076382785537} {CheckpointTs:450115089516462086 MinTableBarrierTs:450115089791713281 AdminJobType:noop} span: {table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}, resolvedTs: 450115089516462087, checkpointTs: 450115089516462086, state: Replicating span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 450115089516462087, checkpointTs: 450115089516462086, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115089516462087, checkpointTs: 450115089516462086, state: Replicating span: {table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}, resolvedTs: 450115089516462087, checkpointTs: 450115089516462086, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450115089516462087, checkpointTs: 450115089516462086, state: Replicating span: {table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}, resolvedTs: 450115089516462087, checkpointTs: 450115089516462086, state: Replicating span: {table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}, resolvedTs: 450115089516462087, checkpointTs: 450115089516462086, state: Replicating span: {table_id:111,start_key:7480000000000000ff6f5f720000000000fa,end_key:7480000000000000ff6f5f730000000000fa}, resolvedTs: 450115089516462087, checkpointTs: 450115089516462086, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/76c84694-c0ae-4b57-9564-49d8bb27547a {"id":"76c84694-c0ae-4b57-9564-49d8bb27547a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc85176405f 76c84694-c0ae-4b57-9564-49d8bb27547a /tidb/cdc/default/default/changefeed/info/9345ff97-0246-4c44-8259-0f8f82516588 {"upstream-id":7374685343960551859,"namespace":"default","changefeed-id":"9345ff97-0246-4c44-8259-0f8f82516588","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T15:05:51.102268174+08:00","start-ts":450115076356833283,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/","use-file-backend":false,"compression":"","flush-concurrency":2,"memory-usage":+ grep -q 'etcd info' {"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115076382785537} /tidb/cdc/default/default/changefeed/status/9345ff97-0246-4c44-8259-0f8f82516588 {"checkpoint-ts":450115089516462086,"min-table-barrier-ts":450115089791713281,"admin-job-type":0} /tidb/cdc/default/default/task/position/76c84694-c0ae-4b57-9564-49d8bb27547a/9345ff97-0246-4c44-8259-0f8f82516588 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374685343960551859 {"id":7374685343960551859,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214a12c00012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:17696, start at 2024-05-30 15:06:42.253388334 +0800 CST m=+5.615937976 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:08:42.264 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:06:42.224 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:56:42.224 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214a12c00012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:17696, start at 2024-05-30 15:06:42.253388334 +0800 CST m=+5.615937976 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:08:42.264 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:06:42.224 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:56:42.224 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214a1434000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:17770, start at 2024-05-30 15:06:42.330108151 +0800 CST m=+5.602983360 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:08:42.339 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:06:42.317 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:56:42.317 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214a13000015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l2n7g, pid:22718, start at 2024-05-30 15:06:42.286398679 +0800 CST m=+5.257122429 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:08:42.293 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:06:42.290 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:56:42.290 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.18996.out cli tso query --pd=http://127.0.0.1:2379 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214a13000015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l2n7g, pid:22718, start at 2024-05-30 15:06:42.286398679 +0800 CST m=+5.257122429 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:08:42.293 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:06:42.290 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:56:42.290 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214a15380008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l2n7g, pid:22801, start at 2024-05-30 15:06:42.390349752 +0800 CST m=+5.307789361 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:08:42.397 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:06:42.382 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:56:42.382 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + set +x + tso='450115090677497857 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115090677497857 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=450115090677497857 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Thu May 30 15:06:47 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.1902719029.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.cli.24117.out cli tso query --pd=http://127.0.0.1:2379 \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } + set +x + tso='450115091219087361 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115091219087361 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x try a VALID cdc server command [Thu May 30 15:06:49 CST 2024] <<<<<< START cdc server in cdc_server_tips case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.2415624158.out server --log-file /tmp/tidb_cdc_test/cdc_server_tips/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cdc_server_tips/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:06:50 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9a7c99c7-3322-45f0-9cff-7b455810ab9f {"id":"9a7c99c7-3322-45f0-9cff-7b455810ab9f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8525ccfbe 9a7c99c7-3322-45f0-9cff-7b455810ab9f /tidb/cdc/default/default/upstream/7374685592610142661 {"id":7374685592610142661,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9a7c99c7-3322-45f0-9cff-7b455810ab9f {"id":"9a7c99c7-3322-45f0-9cff-7b455810ab9f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8525ccfbe 9a7c99c7-3322-45f0-9cff-7b455810ab9f /tidb/cdc/default/default/upstream/7374685592610142661 {"id":7374685592610142661,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9a7c99c7-3322-45f0-9cff-7b455810ab9f {"id":"9a7c99c7-3322-45f0-9cff-7b455810ab9f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8525ccfbe 9a7c99c7-3322-45f0-9cff-7b455810ab9f /tidb/cdc/default/default/upstream/7374685592610142661 {"id":7374685592610142661,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=450115090677497857 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.19071.out cli changefeed create --start-ts=450115090677497857 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7374685592610142661,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T15:06:50.623022017+08:00","start_ts":450115090677497857,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115090677497857,"checkpoint_ts":450115090677497857,"checkpoint_time":"2024-05-30 15:06:45.624"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:06:52 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1a13920b-b9c4-4f30-b3ec-87219f482f04 {"id":"1a13920b-b9c4-4f30-b3ec-87219f482f04","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8526068d0 1a13920b-b9c4-4f30-b3ec-87219f482f04 /tidb/cdc/default/default/upstream/7374685591666776867 {"id":7374685591666776867,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1a13920b-b9c4-4f30-b3ec-87219f482f04 {"id":"1a13920b-b9c4-4f30-b3ec-87219f482f04","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8526068d0 1a13920b-b9c4-4f30-b3ec-87219f482f04 /tidb/cdc/default/default/upstream/7374685591666776867 {"id":7374685591666776867,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1a13920b-b9c4-4f30-b3ec-87219f482f04 {"id":"1a13920b-b9c4-4f30-b3ec-87219f482f04","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8526068d0 1a13920b-b9c4-4f30-b3ec-87219f482f04 /tidb/cdc/default/default/upstream/7374685591666776867 {"id":7374685591666776867,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table test.t1 exists + sleep 5 + kill_tikv ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 17075 19.4 0.4 3777672 1658544 ? Sl 15:06 0:04 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20160 --status-addr 127.0.0.1:20181 --log-file /tmp/tidb_cdc_test/synced_status/tikv1.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv1 jenkins 17076 29.1 0.4 3815040 1730060 ? Sl 15:06 0:07 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20161 --status-addr 127.0.0.1:20182 --log-file /tmp/tidb_cdc_test/synced_status/tikv2.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv2 jenkins 17077 19.4 0.4 3767432 1662904 ? Sl 15:06 0:04 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20162 --status-addr 127.0.0.1:20183 --log-file /tmp/tidb_cdc_test/synced_status/tikv3.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv3 jenkins 17079 28.0 0.4 3798664 1721188 ? Sl 15:06 0:06 tikv-server --pd 127.0.0.1:2479 -A 127.0.0.1:21160 --status-addr 127.0.0.1:21180 --log-file /tmp/tidb_cdc_test/synced_status/tikv_down.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv_down' ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 2225 0 --:--:-- --:--:-- --:--:-- 2229 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-30 15:06:58.724","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"2024-05-30 15:06:52.173","now_ts":"2024-05-30 15:06:59.000","info":"The data syncing is not finished, please wait"}' ++ jq .synced ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:06:58.724","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-05-30' '15:06:52.173","now_ts":"2024-05-30' '15:06:59.000","info":"The' data syncing is not finished, please 'wait"}' + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:06:58.724","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-05-30' '15:06:52.173","now_ts":"2024-05-30' '15:06:59.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 check_redo_resolved_ts 9345ff97-0246-4c44-8259-0f8f82516588 450115095729274884 s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/ /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc_data/redo/9345ff97-0246-4c44-8259-0f8f82516588/meta global resolved ts 450115095309844491 not forward to 450115095729274884 run task failed 1-th time, retry later check_redo_resolved_ts 9345ff97-0246-4c44-8259-0f8f82516588 450115095729274884 s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/ /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc_data/redo/9345ff97-0246-4c44-8259-0f8f82516588/meta run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [2024/05/30 15:07:09.482 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2024/05/30 15:07:09.483 +08:00] [INFO] [apply.go:78] ["set memory limit"] [memoryLimit=10737418240] [2024/05/30 15:07:09.483 +08:00] [INFO] [apply.go:94] ["Start http pprof server"] [addr=:6060] [2024/05/30 15:07:09.484 +08:00] [INFO] [s3.go:406] ["succeed to get bucket region from s3"] ["bucket region"=us-east-1] [2024/05/30 15:07:09.489 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2024/05/30 15:07:09.489 +08:00] [INFO] [redo.go:167] ["apply redo log starts"] [checkpointTs=450115090053857289] [resolvedTs=450115095834132484] [2024/05/30 15:07:09.490 +08:00] [INFO] [s3.go:406] ["succeed to get bucket region from s3"] ["bucket region"=us-east-1] [2024/05/30 15:07:09.490 +08:00] [INFO] [tz.go:36] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/05/30 15:07:09.490 +08:00] [WARN] [config.go:394] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/05/30 15:07:09.490 +08:00] [ERROR] [config.go:209] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2024/05/30 15:07:09.490 +08:00] [INFO] [s3.go:406] ["succeed to get bucket region from s3"] ["bucket region"=us-east-1] [2024/05/30 15:07:09.495 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=5.869189ms] [2024/05/30 15:07:09.496 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22ONLY_FULL_GROUP_BY%2CNO_AUTO_VALUE_ON_ZERO%2CALLOW_INVALID_DATES%2CNO_ENGINE_SUBSTITUTION%2CIGNORE_SPACE%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/05/30 15:07:09.499 +08:00] [INFO] [mysql.go:198] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2024/05/30 15:07:09.500 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:07:09.500 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:07:09.500 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:07:09.500 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:07:09.500 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:07:09.500 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:07:09.500 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:07:09.500 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:07:09.500 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:07:09.501 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:07:09.501 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:07:09.501 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:07:09.501 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:07:09.501 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:07:09.501 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:07:09.501 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:07:09.501 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2024/05/30 15:07:09.501 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2024/05/30 15:07:09.501 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2024/05/30 15:07:09.501 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2024/05/30 15:07:09.501 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2024/05/30 15:07:09.501 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2024/05/30 15:07:09.501 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2024/05/30 15:07:09.501 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2024/05/30 15:07:09.501 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2024/05/30 15:07:09.501 +08:00] [INFO] [tz.go:36] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/05/30 15:07:09.501 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2024/05/30 15:07:09.501 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2024/05/30 15:07:09.501 +08:00] [WARN] [config.go:394] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/05/30 15:07:09.501 +08:00] [ERROR] [config.go:209] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2024/05/30 15:07:09.501 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2024/05/30 15:07:09.501 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2024/05/30 15:07:09.501 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2024/05/30 15:07:09.501 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2024/05/30 15:07:09.501 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2024/05/30 15:07:09.506 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22IGNORE_SPACE%2CNO_AUTO_VALUE_ON_ZERO%2CONLY_FULL_GROUP_BY%2CALLOW_INVALID_DATES%2CNO_ENGINE_SUBSTITUTION%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/05/30 15:07:09.508 +08:00] [INFO] [mysql_ddl_sink.go:99] ["MySQL DDL sink is created"] [namespace=] [changefeed=] [2024/05/30 15:07:09.984 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=495.17446ms] [2024/05/30 15:07:09.985 +08:00] [ERROR] [redo.go:273] ["ignore unsupported DDL"] [ddl="{\"StartTs\":450115090040750096,\"CommitTs\":450115090053857289,\"Query\":\"ALTER TABLE `consistent_replicate_storage_s3`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `consistent_replicate_storage_s3`.`t2`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_storage_s3\",\"tbl-name\":\"t1\",\"tbl-id\":108,\"is-partition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":42,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [2024/05/30 15:07:09.985 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=281] [newQuota=71936] [2024/05/30 15:07:09.998 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=289] [newQuota=73984] [2024/05/30 15:07:10.012 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=289] [newQuota=73984] [2024/05/30 15:07:10.025 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=289] [newQuota=73984] [2024/05/30 15:07:10.043 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=289] [newQuota=73984] [2024/05/30 15:07:10.055 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=289] [newQuota=73984] [2024/05/30 15:07:10.122 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=289] [newQuota=73984] [2024/05/30 15:07:10.137 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=289] [newQuota=73984] [2024/05/30 15:07:10.149 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=289] [newQuota=73984] [2024/05/30 15:07:10.162 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=289] [newQuota=73984] [2024/05/30 15:07:10.231 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=289] [newQuota=73984] [2024/05/30 15:07:10.245 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=289] [newQuota=73984] [2024/05/30 15:07:10.259 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=289] [newQuota=73984] [2024/05/30 15:07:10.333 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=289] [newQuota=73984] [2024/05/30 15:07:10.354 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=289] [newQuota=73984] [2024/05/30 15:07:10.375 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=289] [newQuota=73984] [2024/05/30 15:07:10.428 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=289] [newQuota=73984] [2024/05/30 15:07:10.447 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=289] [newQuota=73984] [2024/05/30 15:07:10.468 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=289] [newQuota=73984] [2024/05/30 15:07:10.527 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=289] [newQuota=73984] [2024/05/30 15:07:10.535 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}] [checkpointTs=450115095834132484] [2024/05/30 15:07:10.535 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}] [checkpointTs=450115095834132484] [2024/05/30 15:07:10.635 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}] [checkpointTs=450115095834132484] [2024/05/30 15:07:10.635 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}] [checkpointTs=450115095834132484] [2024/05/30 15:07:11.538 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:118,start_key:7480000000000000ff765f720000000000fa,end_key:7480000000000000ff765f730000000000fa}] [checkpointTs=450115095834132484] [2024/05/30 15:07:11.538 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:118,start_key:7480000000000000ff765f720000000000fa,end_key:7480000000000000ff765f730000000000fa}] [checkpointTs=450115095834132484] [2024/05/30 15:07:11.639 +08:00] [INFO] [redo.go:223] ["apply redo log finishes"] [appliedLogCount=5008] [appliedDDLCount=0] [currentCheckpoint=450115095834132484] [2024/05/30 15:07:11.639 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] [2024/05/30 15:07:11.639 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2024/05/30 15:07:11.639 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2024/05/30 15:07:11.639 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2024/05/30 15:07:11.639 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2024/05/30 15:07:11.639 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2024/05/30 15:07:11.639 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2024/05/30 15:07:11.639 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2024/05/30 15:07:11.639 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2024/05/30 15:07:11.639 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] [2024/05/30 15:07:11.639 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2024/05/30 15:07:11.639 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2024/05/30 15:07:11.639 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2024/05/30 15:07:11.639 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2024/05/30 15:07:11.639 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2024/05/30 15:07:11.639 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] Apply redo log successfully check diff successfully [Thu May 30 15:07:12 CST 2024] <<<<<< run test case consistent_replicate_storage_s3 success! >>>>>> count(*) 5000 Exiting on signal: INTERRUPT valid ~~~ running cdc Failed to start cdc, the usage tips should be printed 1st test case cdc_server_tips success! try an INVALID cdc server command [Thu May 30 15:07:12 CST 2024] <<<<<< START cdc server in cdc_server_tips case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ true != \n\o ]] + GO_FAILPOINTS= + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.2423724239.out server --log-file /tmp/tidb_cdc_test/cdc_server_tips/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cdc_server_tips/cdc_data --cluster-id default --pd None =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/consistent_partition_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/consistent_partition_table Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) invalid ~~~ running cdc Failed to start cdc, the usage tips should be printed 2nd test case cdc_server_tips success! [Thu May 30 15:07:32 CST 2024] <<<<<< run all test cases cdc_server_tips success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214d6a90000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3, pid:18745, start at 2024-05-30 15:07:37.008603061 +0800 CST m=+5.017675710 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:09:37.014 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:07:36.996 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:57:36.996 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214d6a90000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3, pid:18745, start at 2024-05-30 15:07:37.008603061 +0800 CST m=+5.017675710 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:09:37.014 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:07:36.996 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:57:36.996 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214d6c0c0006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-6ddq3, pid:18819, start at 2024-05-30 15:07:37.097029824 +0800 CST m=+5.053077550 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:09:37.102 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:07:37.091 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:57:37.091 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_partition_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_partition_table/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/consistent_partition_table/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/consistent_partition_table/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_partition_table/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Thu May 30 15:07:40 CST 2024] <<<<<< START cdc server in consistent_partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_partition_table.2008420086.out server --log-file /tmp/tidb_cdc_test/consistent_partition_table/cdcpartition_table.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_partition_table/cdc_datapartition_table.server1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:07:43 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3938c2e3-1c89-40a1-ba61-1184e827830c {"id":"3938c2e3-1c89-40a1-ba61-1184e827830c","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8532d80ba 3938c2e3-1c89-40a1-ba61-1184e827830c /tidb/cdc/default/default/upstream/7374685829245227212 {"id":7374685829245227212,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3938c2e3-1c89-40a1-ba61-1184e827830c {"id":"3938c2e3-1c89-40a1-ba61-1184e827830c","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8532d80ba 3938c2e3-1c89-40a1-ba61-1184e827830c /tidb/cdc/default/default/upstream/7374685829245227212 {"id":7374685829245227212,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3938c2e3-1c89-40a1-ba61-1184e827830c {"id":"3938c2e3-1c89-40a1-ba61-1184e827830c","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8532d80ba 3938c2e3-1c89-40a1-ba61-1184e827830c /tidb/cdc/default/default/upstream/7374685829245227212 {"id":7374685829245227212,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table partition_table2.t2 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/event_filter/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table partition_table2.t2 not exists for 2-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/event_filter Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... table partition_table2.t2 not exists for 3-th check, retry later table partition_table2.t2 not exists for 4-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table partition_table2.t2 not exists for 5-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table2.t2 not exists for 6-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table2.t2 not exists for 7-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table2.t2 not exists for 8-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table2.t2 not exists for 9-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214eb85c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l2n7g, pid:25188, start at 2024-05-30 15:07:58.38864109 +0800 CST m=+5.387905301 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:09:58.397 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:07:58.359 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:57:58.359 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214eb85c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l2n7g, pid:25188, start at 2024-05-30 15:07:58.38864109 +0800 CST m=+5.387905301 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:09:58.397 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:07:58.359 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:57:58.359 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214eb8440015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-l2n7g, pid:25264, start at 2024-05-30 15:07:58.401772212 +0800 CST m=+5.344600579 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:09:58.408 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:07:58.403 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:57:58.403 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/event_filter/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/event_filter/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table partition_table2.t2 exists [Thu May 30 15:08:03 CST 2024] <<<<<< START cdc server in event_filter case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.event_filter.2655626558.out server --log-file /tmp/tidb_cdc_test/event_filter/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/event_filter/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:08:06 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7e9e34c9-079e-473f-83d7-b641408e79d8 {"id":"7e9e34c9-079e-473f-83d7-b641408e79d8","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8538158bc 7e9e34c9-079e-473f-83d7-b641408e79d8 /tidb/cdc/default/default/upstream/7374685918428470148 {"id":7374685918428470148,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7e9e34c9-079e-473f-83d7-b641408e79d8 {"id":"7e9e34c9-079e-473f-83d7-b641408e79d8","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8538158bc 7e9e34c9-079e-473f-83d7-b641408e79d8 /tidb/cdc/default/default/upstream/7374685918428470148 {"id":7374685918428470148,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7e9e34c9-079e-473f-83d7-b641408e79d8 {"id":"7e9e34c9-079e-473f-83d7-b641408e79d8","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8538158bc 7e9e34c9-079e-473f-83d7-b641408e79d8 /tidb/cdc/default/default/upstream/7374685918428470148 {"id":7374685918428470148,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.event_filter.cli.26604.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --server=127.0.0.1:8300 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/event_filter/conf/cf.toml Create changefeed successfully! ID: 9befd398-03c6-4299-8f7f-686451af15f4 Info: {"upstream_id":7374685918428470148,"namespace":"default","id":"9befd398-03c6-4299-8f7f-686451af15f4","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T15:08:06.843597628+08:00","start_ts":450115111946289153,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["event_filter.*"],"event_filters":[{"matcher":["event_filter.t1"],"ignore_event":["drop table","delete"],"ignore_sql":null,"ignore_insert_value_expr":"id = 2 or city = 'tokyo'","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""},{"matcher":["event_filter.t_truncate"],"ignore_event":["truncate table"],"ignore_sql":null,"ignore_insert_value_expr":"","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""},{"matcher":["event_filter.t_alter"],"ignore_event":["alter table"],"ignore_sql":null,"ignore_insert_value_expr":"","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""}]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115111946289153,"checkpoint_ts":450115111946289153,"checkpoint_time":"2024-05-30 15:08:06.758"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... wait process cdc.test exit for 1-th time... + set +x wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:08:09 CST 2024] <<<<<< START cdc server in consistent_partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=return(true);github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql/MySQLSinkExecDDLDelay=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_partition_table.2026220264.out server --log-file /tmp/tidb_cdc_test/consistent_partition_table/cdcpartition_table.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_partition_table/cdc_datapartition_table.server2 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table event_filter.t1 does not exists table event_filter.t1 exists table event_filter.t_normal not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:08:12 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/9b05f12f-95d0-44f6-88cf-b9f5728c36d2 {UpstreamID:7374685829245227212 Namespace:default ID:9b05f12f-95d0-44f6-88cf-b9f5728c36d2 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 15:07:43.670440835 +0800 CST StartTs:450115105861664771 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003640360 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115105887617029} {CheckpointTs:450115111497498629 MinTableBarrierTs:450115113332506630 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:105,start_key:7480000000000000ff695f720000000000fa,end_key:7480000000000000ff695f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:119,start_key:7480000000000000ff775f720000000000fa,end_key:7480000000000000ff775f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:123,start_key:7480000000000000ff7b5f720000000000fa,end_key:7480000000000000ff7b5f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:107,start_key:7480000000000000ff6b5f720000000000fa,end_key:7480000000000000ff6b5f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:117,start_key:7480000000000000ff755f720000000000fa,end_key:7480000000000000ff755f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7a6a3c67-8667-4195-a635-b111c9df7a9e {"id":"7a6a3c67-8667-4195-a635-b111c9df7a9e","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8532d81e8 7a6a3c67-8667-4195-a635-b111c9df7a9e /tidb/cdc/default/default/changefeed/info/9b05f12f-95d0-44f6-88cf-b9f5728c36d2 {"upstream-id":7374685829245227212,"namespace":"default","changefeed-id":"9b05f12f-95d0-44f6-88cf-b9f5728c36d2","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T15:07:43.670440835+08:00","start-ts":450115105861664771,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_partition_table/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115105887617029} /tidb/cdc/default/default/changefeed/status/9b05f12f-95d0-44f6-88cf-b9f5728c36d2 {"checkpoint-ts":450115111497498629,"min-table-barrier-ts":450115113332506630,"admin-job-type":0} /tidb/cdc/default/default/task/position/7a6a3c67-8667-4195-a635-b111c9df7a9e/9b05f12f-95d0-44f6-88cf-b9f5728c36d2 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374685829245227212 {"id":7374685829245227212,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/9b05f12f-95d0-44f6-88cf-b9f5728c36d2 {UpstreamID:7374685829245227212 Namespace:default ID:9b05f12f-95d0-44f6-88cf-b9f5728c36d2 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 15:07:43.670440835 +0800 CST StartTs:450115105861664771 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003640360 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115105887617029} {CheckpointTs:450115111497498629 MinTableBarrierTs:450115113332506630 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:105,start_key:7480000000000000ff695f720000000000fa,end_key:7480000000000000ff695f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:119,start_key:7480000000000000ff775f720000000000fa,end_key:7480000000000000ff775f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:123,start_key:7480000000000000ff7b5f720000000000fa,end_key:7480000000000000ff7b5f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:107,start_key:7480000000000000ff6b5f720000000000fa,end_key:7480000000000000ff6b5f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:117,start_key:7480000000000000ff755f720000000000fa,end_key:7480000000000000ff755f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7a6a3c67-8667-4195-a635-b111c9df7a9e {"id":"7a6a3c67-8667-4195-a635-b111c9df7a9e","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8532d81e8 7a6a3c67-8667-4195-a635-b111c9df7a9e /tidb/cdc/default/default/changefeed/info/9b05f12f-95d0-44f6-88cf-b9f5728c36d2 {"upstream-id":7374685829245227212,"namespace":"default","changefeed-id":"9b05f12f-95d0-44f6-88cf-b9f5728c36d2","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T15:07:43.670440835+08:00","start-ts":450115105861664771,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_partition_table/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115105887617029} /tidb/cdc/default/default/changefeed/status/9b05f12f-95d0-44f6-88cf-b9f5728c36d2 {"checkpoint-ts":450115111497498629,"min-table-barrier-ts":450115113332506630,"admin-job-type":0} /tidb/cdc/default/default/task/position/7a6a3c67-8667-4195-a635-b111c9df7a9e/9b05f12f-95d0-44f6-88cf-b9f5728c36d2 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374685829245227212 {"id":7374685829245227212,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/9b05f12f-95d0-44f6-88cf-b9f5728c36d2 {UpstreamID:7374685829245227212 Namespace:default ID:9b05f12f-95d0-44f6-88cf-b9f5728c36d2 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-05-30 15:07:43.670440835 +0800 CST StartTs:450115105861664771 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003640360 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115105887617029} {CheckpointTs:450115111497498629 MinTableBarrierTs:450115113332506630 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:105,start_key:7480000000000000ff695f720000000000fa,end_key:7480000000000000ff695f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:119,start_key:7480000000000000ff775f720000000000fa,end_key:7480000000000000ff775f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:123,start_key:7480000000000000ff7b5f720000000000fa,end_key:7480000000000000ff7b5f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:107,start_key:7480000000000000ff6b5f720000000000fa,end_key:7480000000000000ff6b5f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating span: {table_id:117,start_key:7480000000000000ff755f720000000000fa,end_key:7480000000000000ff755f730000000000fa}, resolvedTs: 450115111497498629, checkpointTs: 450115111497498629, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7a6a3c67-8667-4195-a635-b111c9df7a9e {"id":"7a6a3c67-8667-4195-a635-b111c9df7a9e","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8532d81e8 7a6a3c67-8667-4195-a635-b111c9df7a9e /tidb/cdc/default/default/changefeed/info/9b05f12f-95d0-44f6-88cf-b9f5728c36d2 {"upstream-id":7374685829245227212,"namespace":"default","changefeed-id":"9b05f12f-95d0-44f6-88cf-b9f5728c36d2","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-05-30T15:07:43.670440835+08:00","start-ts":450115105861664771,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_partition_table/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115105887617029} /tidb/cdc/default/default/changefeed/status/9b05f12f-95d0-44f6-88cf-b9f5728c36d2 {"checkpoint-ts":450115111497498629,"min-table-barrier-ts":450115113332506630,"admin-job-type":0} /tidb/cdc/default/default/task/position/7a6a3c67-8667-4195-a635-b111c9df7a9e/9b05f12f-95d0-44f6-88cf-b9f5728c36d2 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374685829245227212 {"id":7374685829245227212,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x table event_filter.t_normal exists table event_filter.t_truncate not exists for 1-th check, retry later table event_filter.t_truncate exists table event_filter.t_alter not exists for 1-th check, retry later table event_filter.t_alter exists table event_filter.finish_mark exists check diff failed 1-th time, retry later check diff failed 2-th time, retry later check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 15:08:25 CST 2024] <<<<<< run test case event_filter success! >>>>>> \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 9538 0 --:--:-- --:--:-- --:--:-- 9640 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-30 15:06:58.724","puller_resolved_ts":"2024-05-30 15:06:58.724","last_synced_ts":"2024-05-30 15:06:52.173","now_ts":"2024-05-30 15:09:09.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:06:58.724","puller_resolved_ts":"2024-05-30' '15:06:58.724","last_synced_ts":"2024-05-30' '15:06:52.173","now_ts":"2024-05-30' '15:09:09.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ jq -r .info ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:06:58.724","puller_resolved_ts":"2024-05-30' '15:06:58.724","last_synced_ts":"2024-05-30' '15:06:52.173","now_ts":"2024-05-30' '15:09:09.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tidb conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21547dec000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:20094, start at 2024-05-30 15:09:32.938531211 +0800 CST m=+5.655165122 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:11:32.948 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:09:32.923 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:59:32.923 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21547dec000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:20094, start at 2024-05-30 15:09:32.938531211 +0800 CST m=+5.655165122 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:11:32.948 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:09:32.923 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:59:32.923 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21547eac0017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:20160, start at 2024-05-30 15:09:33.01782747 +0800 CST m=+5.650678741 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:11:33.027 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:09:33.021 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:59:33.021 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.21515.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='450115135870861313 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115135870861313 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=450115135870861313 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Thu May 30 15:09:40 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2154621548.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:09:43 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/31024095-dbb3-4161-b077-370d33d7c18d {"id":"31024095-dbb3-4161-b077-370d33d7c18d","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc854f576c7 31024095-dbb3-4161-b077-370d33d7c18d /tidb/cdc/default/default/upstream/7374686327590066423 {"id":7374686327590066423,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/31024095-dbb3-4161-b077-370d33d7c18d {"id":"31024095-dbb3-4161-b077-370d33d7c18d","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc854f576c7 31024095-dbb3-4161-b077-370d33d7c18d /tidb/cdc/default/default/upstream/7374686327590066423 {"id":7374686327590066423,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/31024095-dbb3-4161-b077-370d33d7c18d {"id":"31024095-dbb3-4161-b077-370d33d7c18d","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc854f576c7 31024095-dbb3-4161-b077-370d33d7c18d /tidb/cdc/default/default/upstream/7374686327590066423 {"id":7374686327590066423,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=450115135870861313 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.21602.out cli changefeed create --start-ts=450115135870861313 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7374686327590066423,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T15:09:44.024511687+08:00","start_ts":450115135870861313,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115135870861313,"checkpoint_ts":450115135870861313,"checkpoint_time":"2024-05-30 15:09:38.023"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 + kill_tidb ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 20094 12.6 0.0 2254468 208964 ? Sl 15:09 0:03 tidb-server -P 4000 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1717052967274311853.toml --store tikv --path 127.0.0.1:2379 --status=10080 --log-file /tmp/tidb_cdc_test/synced_status/tidb.log jenkins 20098 4.2 0.0 2425592 164020 ? Sl 15:09 0:01 tidb-server -P 4001 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1717052967278693866.toml --store tikv --path 127.0.0.1:2379 --status=10081 --log-file /tmp/tidb_cdc_test/synced_status/tidb_other.log jenkins 20160 12.8 0.0 2427352 219240 ? Sl 15:09 0:03 tidb-server -P 3306 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1717052967355834536.toml --store tikv --path 127.0.0.1:2479 --status=20080 --log-file /tmp/tidb_cdc_test/synced_status/tidb_down.log' ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 2546 0 --:--:-- --:--:-- --:--:-- 2557 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-30 15:09:51.374","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"2024-05-30 15:09:45.573","now_ts":"2024-05-30 15:09:52.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:09:51.374","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-05-30' '15:09:45.573","now_ts":"2024-05-30' '15:09:52.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:09:51.374","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-05-30' '15:09:45.573","now_ts":"2024-05-30' '15:09:52.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit # diff Configuration. check-thread-count = 4 export-fix-sql = true check-struct-only = false [task] output-dir = "/tmp/tidb_cdc_test/partition_table/sync_diff/output" source-instances = ["mysql1"] target-instance = "tidb0" target-check-tables = ["partition_table.?*"] [data-sources] [data-sources.mysql1] host = "127.0.0.1" port = 4000 user = "root" password = "" [data-sources.tidb0] host = "127.0.0.1" port = 3306 user = "root" password = "" [2024/05/30 15:10:16.360 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2024/05/30 15:10:16.361 +08:00] [INFO] [apply.go:78] ["set memory limit"] [memoryLimit=10737418240] [2024/05/30 15:10:16.361 +08:00] [INFO] [apply.go:94] ["Start http pprof server"] [addr=:6060] [2024/05/30 15:10:16.362 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2024/05/30 15:10:16.362 +08:00] [INFO] [redo.go:167] ["apply redo log starts"] [checkpointTs=450115111497498629] [resolvedTs=450115145091252228] [2024/05/30 15:10:16.363 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=1.047848ms] [2024/05/30 15:10:16.363 +08:00] [INFO] [tz.go:36] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/05/30 15:10:16.363 +08:00] [WARN] [config.go:394] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/05/30 15:10:16.363 +08:00] [ERROR] [config.go:209] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2024/05/30 15:10:16.363 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=1.109354ms] [2024/05/30 15:10:16.367 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22NO_ENGINE_SUBSTITUTION%2CALLOW_INVALID_DATES%2CIGNORE_SPACE%2CONLY_FULL_GROUP_BY%2CNO_AUTO_VALUE_ON_ZERO%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/05/30 15:10:16.375 +08:00] [INFO] [mysql.go:198] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2024/05/30 15:10:16.377 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:10:16.377 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:10:16.377 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:10:16.377 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:10:16.377 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:10:16.377 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:10:16.377 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:10:16.377 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:10:16.377 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:10:16.377 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:10:16.377 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:10:16.377 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:10:16.377 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:10:16.377 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:10:16.377 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:10:16.377 +08:00] [INFO] [txn_cache.go:64] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitAll] [2024/05/30 15:10:16.377 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2024/05/30 15:10:16.377 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2024/05/30 15:10:16.377 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2024/05/30 15:10:16.377 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2024/05/30 15:10:16.377 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2024/05/30 15:10:16.377 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2024/05/30 15:10:16.377 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2024/05/30 15:10:16.377 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2024/05/30 15:10:16.377 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2024/05/30 15:10:16.377 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2024/05/30 15:10:16.377 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2024/05/30 15:10:16.377 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2024/05/30 15:10:16.377 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2024/05/30 15:10:16.377 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2024/05/30 15:10:16.377 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2024/05/30 15:10:16.377 +08:00] [INFO] [worker.go:86] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2024/05/30 15:10:16.377 +08:00] [INFO] [tz.go:36] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/05/30 15:10:16.377 +08:00] [WARN] [config.go:394] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/05/30 15:10:16.377 +08:00] [ERROR] [config.go:209] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2024/05/30 15:10:16.381 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22NO_ENGINE_SUBSTITUTION%2CALLOW_INVALID_DATES%2CIGNORE_SPACE%2CONLY_FULL_GROUP_BY%2CNO_AUTO_VALUE_ON_ZERO%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/05/30 15:10:16.383 +08:00] [INFO] [mysql_ddl_sink.go:99] ["MySQL DDL sink is created"] [namespace=] [changefeed=] [2024/05/30 15:10:16.383 +08:00] [INFO] [redo.go:263] ["reset quota"] [rowSize=264] [newQuota=67584] [2024/05/30 15:10:16.383 +08:00] [WARN] [redo.go:288] ["apply DDL"] [ddl="{\"StartTs\":450115113516007428,\"CommitTs\":450115113516007433,\"Query\":\"alter table t1 drop partition p1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"partition_table\",\"tbl-name\":\"t1\",\"tbl-id\":111,\"is-partition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":20,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [2024/05/30 15:10:18.390 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":450115113516007428,\"CommitTs\":450115113516007433,\"Query\":\"alter table t1 drop partition p1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"partition_table\",\"tbl-name\":\"t1\",\"tbl-id\":111,\"is-partition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":20,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [namespace=] [changefeed=] [2024/05/30 15:10:18.599 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="alter table t1 drop partition p1"] [duration=208.645831ms] [namespace=] [changefeed=] [2024/05/30 15:10:18.599 +08:00] [WARN] [redo.go:288] ["apply DDL"] [ddl="{\"StartTs\":450115113555329031,\"CommitTs\":450115113555329037,\"Query\":\"ALTER TABLE `partition_table`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `partition_table`.`t2`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"partition_table\",\"tbl-name\":\"t1\",\"tbl-id\":111,\"is-partition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":42,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [2024/05/30 15:10:18.800 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":450115113555329031,\"CommitTs\":450115113555329037,\"Query\":\"ALTER TABLE `partition_table`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `partition_table`.`t2`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"partition_table\",\"tbl-name\":\"t1\",\"tbl-id\":111,\"is-partition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":42,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [namespace=] [changefeed=] [2024/05/30 15:10:18.951 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="ALTER TABLE `partition_table`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `partition_table`.`t2`"] [duration=151.129951ms] [namespace=] [changefeed=] [2024/05/30 15:10:18.951 +08:00] [WARN] [redo.go:288] ["apply DDL"] [ddl="{\"StartTs\":450115113594650632,\"CommitTs\":450115113594650637,\"Query\":\"ALTER TABLE `partition_table`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `partition_table2`.`t2`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"partition_table\",\"tbl-name\":\"t1\",\"tbl-id\":111,\"is-partition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":42,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [2024/05/30 15:10:19.353 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":450115113594650632,\"CommitTs\":450115113594650637,\"Query\":\"ALTER TABLE `partition_table`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `partition_table2`.`t2`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"partition_table\",\"tbl-name\":\"t1\",\"tbl-id\":111,\"is-partition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":42,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [namespace=] [changefeed=] [2024/05/30 15:10:19.499 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="ALTER TABLE `partition_table`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `partition_table2`.`t2`"] [duration=146.306136ms] [namespace=] [changefeed=] [2024/05/30 15:10:19.499 +08:00] [WARN] [redo.go:288] ["apply DDL"] [ddl="{\"StartTs\":450115113856794633,\"CommitTs\":450115113856794637,\"Query\":\"ALTER TABLE t1 REORGANIZE PARTITION p0,p2 INTO (PARTITION p0 VALUES LESS THAN (5), PARTITION p1 VALUES LESS THAN (10), PARTITION p2 VALUES LESS THAN (21))\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"partition_table\",\"tbl-name\":\"t1\",\"tbl-id\":111,\"is-partition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":64,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [2024/05/30 15:10:19.901 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":450115113856794633,\"CommitTs\":450115113856794637,\"Query\":\"ALTER TABLE t1 REORGANIZE PARTITION p0,p2 INTO (PARTITION p0 VALUES LESS THAN (5), PARTITION p1 VALUES LESS THAN (10), PARTITION p2 VALUES LESS THAN (21))\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"partition_table\",\"tbl-name\":\"t1\",\"tbl-id\":111,\"is-partition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":64,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false}"] [namespace=] [changefeed=] [2024/05/30 15:10:21.070 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="ALTER TABLE t1 REORGANIZE PARTITION p0,p2 INTO (PARTITION p0 VALUES LESS THAN (5), PARTITION p1 VALUES LESS THAN (10), PARTITION p2 VALUES LESS THAN (21))"] [duration=1.168774119s] [namespace=] [changefeed=] [2024/05/30 15:10:21.070 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}] [checkpointTs=450115145091252228] [2024/05/30 15:10:21.070 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}] [checkpointTs=450115145091252228] [2024/05/30 15:10:21.171 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}] [checkpointTs=450115145091252228] [2024/05/30 15:10:21.171 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}] [checkpointTs=450115145091252228] [2024/05/30 15:10:21.272 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}] [checkpointTs=450115145091252228] [2024/05/30 15:10:21.272 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}] [checkpointTs=450115145091252228] [2024/05/30 15:10:21.373 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:117,start_key:7480000000000000ff755f720000000000fa,end_key:7480000000000000ff755f730000000000fa}] [checkpointTs=450115145091252228] [2024/05/30 15:10:21.373 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:117,start_key:7480000000000000ff755f720000000000fa,end_key:7480000000000000ff755f730000000000fa}] [checkpointTs=450115145091252228] [2024/05/30 15:10:21.473 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:123,start_key:7480000000000000ff7b5f720000000000fa,end_key:7480000000000000ff7b5f730000000000fa}] [checkpointTs=450115145091252228] [2024/05/30 15:10:21.473 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:123,start_key:7480000000000000ff7b5f720000000000fa,end_key:7480000000000000ff7b5f730000000000fa}] [checkpointTs=450115145091252228] [2024/05/30 15:10:21.574 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}] [checkpointTs=450115145091252228] [2024/05/30 15:10:21.574 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}] [checkpointTs=450115145091252228] [2024/05/30 15:10:21.675 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:105,start_key:7480000000000000ff695f720000000000fa,end_key:7480000000000000ff695f730000000000fa}] [checkpointTs=450115145091252228] [2024/05/30 15:10:21.675 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:105,start_key:7480000000000000ff695f720000000000fa,end_key:7480000000000000ff695f730000000000fa}] [checkpointTs=450115145091252228] [2024/05/30 15:10:21.776 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}] [checkpointTs=450115145091252228] [2024/05/30 15:10:21.776 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}] [checkpointTs=450115145091252228] [2024/05/30 15:10:21.877 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:107,start_key:7480000000000000ff6b5f720000000000fa,end_key:7480000000000000ff6b5f730000000000fa}] [checkpointTs=450115145091252228] [2024/05/30 15:10:21.877 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:107,start_key:7480000000000000ff6b5f720000000000fa,end_key:7480000000000000ff6b5f730000000000fa}] [checkpointTs=450115145091252228] [2024/05/30 15:10:21.978 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}] [checkpointTs=450115145091252228] [2024/05/30 15:10:21.978 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}] [checkpointTs=450115145091252228] [2024/05/30 15:10:22.079 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}] [checkpointTs=450115145091252228] [2024/05/30 15:10:22.079 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}] [checkpointTs=450115145091252228] [2024/05/30 15:10:22.180 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:119,start_key:7480000000000000ff775f720000000000fa,end_key:7480000000000000ff775f730000000000fa}] [checkpointTs=450115145091252228] [2024/05/30 15:10:22.180 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:119,start_key:7480000000000000ff775f720000000000fa,end_key:7480000000000000ff775f730000000000fa}] [checkpointTs=450115145091252228] [2024/05/30 15:10:22.281 +08:00] [INFO] [redo.go:223] ["apply redo log finishes"] [appliedLogCount=47] [appliedDDLCount=4] [currentCheckpoint=450115145091252228] [2024/05/30 15:10:22.281 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2024/05/30 15:10:22.281 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2024/05/30 15:10:22.281 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2024/05/30 15:10:22.281 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2024/05/30 15:10:22.281 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2024/05/30 15:10:22.281 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] [2024/05/30 15:10:22.281 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2024/05/30 15:10:22.281 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2024/05/30 15:10:22.281 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2024/05/30 15:10:22.281 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2024/05/30 15:10:22.281 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2024/05/30 15:10:22.281 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2024/05/30 15:10:22.281 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] [2024/05/30 15:10:22.281 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2024/05/30 15:10:22.281 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2024/05/30 15:10:22.281 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] Apply redo log successfully check diff successfully [Thu May 30 15:10:22 CST 2024] <<<<<< run test case consistent_partition_table success! >>>>>> Database INFORMATION_SCHEMA METRICS_SCHEMA PERFORMANCE_SCHEMA mysql partition_table partition_table2 test =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kafka_big_txn_v2/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:10:36 CST 2024] <<<<<< run test case kafka_big_txn_v2 success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kafka_big_messages_v2/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:10:39 CST 2024] <<<<<< run test case kafka_big_messages_v2 success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/multi_tables_ddl_v2/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:10:42 CST 2024] <<<<<< run test case multi_tables_ddl_v2 success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/multi_topics_v2/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:10:50 CST 2024] <<<<<< run test case multi_topics_v2 success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/storage_cleanup/run.sh using Sink-Type: mysql... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/storage_cleanup/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/storage_cleanup ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/storage_cleanup + source /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/storage_cleanup/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/storage_cleanup + CDC_BINARY=cdc.test + SINK_TYPE=mysql + EXIST_FILES=() + CLEANED_FILES=() + trap stop_tidb_cluster EXIT + run mysql + '[' mysql '!=' storage ']' + return + check_logs /tmp/tidb_cdc_test/storage_cleanup ++ date + echo '[Thu May 30 15:10:53 CST 2024] <<<<<< run test case storage_cleanup success! >>>>>>' [Thu May 30 15:10:53 CST 2024] <<<<<< run test case storage_cleanup success! >>>>>> + stop_tidb_cluster =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/csv_storage_basic/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:10:58 CST 2024] <<<<<< run test case csv_storage_basic success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:11:01 CST 2024] <<<<<< run test case csv_storage_multi_tables_ddl success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/csv_storage_partition_table/run.sh using Sink-Type: mysql... <<================= [Thu May 30 15:11:03 CST 2024] <<<<<< run test case csv_storage_partition_table success! >>>>>> \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2799 0 --:--:-- --:--:-- --:--:-- 2833 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-30 15:12:02.474","puller_resolved_ts":"2024-05-30 15:11:54.474","last_synced_ts":"2024-05-30 15:09:45.573","now_ts":"2024-05-30 15:12:02.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:12:02.474","puller_resolved_ts":"2024-05-30' '15:11:54.474","last_synced_ts":"2024-05-30' '15:09:45.573","now_ts":"2024-05-30' '15:12:02.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:12:02.474","puller_resolved_ts":"2024-05-30' '15:11:54.474","last_synced_ts":"2024-05-30' '15:09:45.573","now_ts":"2024-05-30' '15:12:02.000","info":"Data' syncing is 'finished"}' ++ jq -r .info + info='Data syncing is finished' + target_message='Data syncing is finished' + '[' 'Data syncing is finished' '!=' 'Data syncing is finished' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_failpoint conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f215eec740011 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:22718, start at 2024-05-30 15:12:23.856579364 +0800 CST m=+5.284362083 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:14:23.865 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:12:23.837 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:02:23.837 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f215eec740011 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:22718, start at 2024-05-30 15:12:23.856579364 +0800 CST m=+5.284362083 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:14:23.865 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:12:23.837 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:02:23.837 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f215eee000010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:22796, start at 2024-05-30 15:12:23.954534232 +0800 CST m=+5.315080646 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:14:23.963 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:12:23.936 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:02:23.936 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status + export 'GO_FAILPOINTS=github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.24069.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='450115180766429188 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115180766429188 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=450115180766429188 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Thu May 30 15:12:30 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2410324105.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:12:33 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/630f73d0-257e-4192-8a41-06517e5ffa2c {"id":"630f73d0-257e-4192-8a41-06517e5ffa2c","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc857968ec0 630f73d0-257e-4192-8a41-06517e5ffa2c /tidb/cdc/default/default/upstream/7374687061465967220 {"id":7374687061465967220,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/630f73d0-257e-4192-8a41-06517e5ffa2c {"id":"630f73d0-257e-4192-8a41-06517e5ffa2c","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc857968ec0 630f73d0-257e-4192-8a41-06517e5ffa2c /tidb/cdc/default/default/upstream/7374687061465967220 {"id":7374687061465967220,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/630f73d0-257e-4192-8a41-06517e5ffa2c {"id":"630f73d0-257e-4192-8a41-06517e5ffa2c","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc857968ec0 630f73d0-257e-4192-8a41-06517e5ffa2c /tidb/cdc/default/default/upstream/7374687061465967220 {"id":7374687061465967220,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=450115180766429188 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.24147.out cli changefeed create --start-ts=450115180766429188 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7374687061465967220,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T15:12:34.329727792+08:00","start_ts":450115180766429188,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115180766429188,"checkpoint_ts":450115180766429188,"checkpoint_time":"2024-05-30 15:12:29.286"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x + sleep 20 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 5697 0 --:--:-- --:--:-- --:--:-- 5738 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-30 15:12:29.286","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-05-30 15:12:55.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:12:29.286","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-30' '15:12:55.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:12:29.286","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-30' '15:12:55.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + export GO_FAILPOINTS= + GO_FAILPOINTS= + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_normal_case_and_unavailable_pd conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2162d7c40014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:25075, start at 2024-05-30 15:13:28.079442833 +0800 CST m=+5.327647903 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:15:28.087 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:13:28.049 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:03:28.049 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2162d7c40014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:25075, start at 2024-05-30 15:13:28.079442833 +0800 CST m=+5.327647903 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:15:28.087 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:13:28.049 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:03:28.049 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2162d8a00014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:25148, start at 2024-05-30 15:13:28.149665534 +0800 CST m=+5.320663498 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:15:28.157 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:13:28.153 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:03:28.153 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.26456.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='450115197586374657 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115197586374657 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=450115197586374657 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Thu May 30 15:13:34 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2649226494.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:13:38 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c59f5af5-ac16-4bc9-988c-836337ce81d9 {"id":"c59f5af5-ac16-4bc9-988c-836337ce81d9","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8588962c3 c59f5af5-ac16-4bc9-988c-836337ce81d9 /tidb/cdc/default/default/upstream/7374687337216555693 {"id":7374687337216555693,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c59f5af5-ac16-4bc9-988c-836337ce81d9 {"id":"c59f5af5-ac16-4bc9-988c-836337ce81d9","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8588962c3 c59f5af5-ac16-4bc9-988c-836337ce81d9 /tidb/cdc/default/default/upstream/7374687337216555693 {"id":7374687337216555693,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c59f5af5-ac16-4bc9-988c-836337ce81d9 {"id":"c59f5af5-ac16-4bc9-988c-836337ce81d9","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8588962c3 c59f5af5-ac16-4bc9-988c-836337ce81d9 /tidb/cdc/default/default/upstream/7374687337216555693 {"id":7374687337216555693,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=450115197586374657 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.26533.out cli changefeed create --start-ts=450115197586374657 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7374687337216555693,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T15:13:38.520826351+08:00","start_ts":450115197586374657,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115197586374657,"checkpoint_ts":450115197586374657,"checkpoint_time":"2024-05-30 15:13:33.449"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 5514 0 --:--:-- --:--:-- --:--:-- 5666 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-30 15:13:33.449","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-05-30 15:13:39.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:13:33.449","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-30' '15:13:39.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:13:33.449","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-30' '15:13:39.000","info":"Data' syncing is 'finished"}' ++ jq -r .sink_checkpoint_ts + sink_checkpoint_ts='2024-05-30 15:13:33.449' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:13:33.449","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-30' '15:13:39.000","info":"Data' syncing is 'finished"}' ++ jq -r .puller_resolved_ts + puller_resolved_ts='1970-01-01 08:00:00.000' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:13:33.449","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-30' '15:13:39.000","info":"Data' syncing is 'finished"}' ++ jq -r .last_synced_ts + last_synced_ts='1970-01-01 08:00:00.000' + '[' true '!=' true ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' ++ date '+%Y-%m-%d %H:%M:%S' + current='2024-05-30 15:13:40' + echo 'sink_checkpoint_ts is 2024-05-30' 15:13:33.449 sink_checkpoint_ts is 2024-05-30 15:13:33.449 ++ date -d '2024-05-30 15:13:33.449' +%s + checkpoint_timestamp=1717053213 ++ date -d '2024-05-30 15:13:40' +%s + current_timestamp=1717053220 + '[' 7 -gt 300 ']' + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 3992 0 --:--:-- --:--:-- --:--:-- 4050 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-30 15:13:45.799","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"2024-05-30 15:13:40.249","now_ts":"2024-05-30 15:13:47.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:13:45.799","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-05-30' '15:13:40.249","now_ts":"2024-05-30' '15:13:47.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:13:45.799","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-05-30' '15:13:40.249","now_ts":"2024-05-30' '15:13:47.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2719 0 --:--:-- --:--:-- --:--:-- 2728 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-30 15:15:55.899","puller_resolved_ts":"2024-05-30 15:15:49.899","last_synced_ts":"2024-05-30 15:13:40.249","now_ts":"2024-05-30 15:15:57.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:15:55.899","puller_resolved_ts":"2024-05-30' '15:15:49.899","last_synced_ts":"2024-05-30' '15:13:40.249","now_ts":"2024-05-30' '15:15:57.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' + kill_pd ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 24407 8.7 0.0 13502668 141352 ? Sl 15:13 0:14 pd-server --advertise-client-urls http://127.0.0.1:2379 --client-urls http://0.0.0.0:2379 --advertise-peer-urls http://127.0.0.1:2380 --peer-urls http://0.0.0.0:2380 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/pd1.log --data-dir /tmp/tidb_cdc_test/synced_status/pd1 --name=pd1 --initial-cluster=pd1=http://127.0.0.1:2380 jenkins 24458 5.8 0.0 13846676 134708 ? Sl 15:13 0:09 pd-server --advertise-client-urls http://127.0.0.1:2479 --client-urls http://0.0.0.0:2479 --advertise-peer-urls http://127.0.0.1:2480 --peer-urls http://0.0.0.0:2480 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/down_pd.log --data-dir /tmp/tidb_cdc_test/synced_status/down_pd' ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 + sleep 20 {"level":"warn","ts":1717053363.1143444,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000e4afc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1717053363.1144736,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1717053363.1817117,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0039b0e00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1717053363.1818132,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1717053364.030472,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00248e380/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1717053364.03055,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-05-30T15:16:07.840395+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0012ca000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:16:07.841897+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001350000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:16:07.892298+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000919180/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:16:13.842023+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0012ca000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:16:13.842643+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001350000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:16:13.893946+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000919180/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:16:19.843385+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0012ca000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:16:19.84439+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001350000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:16:19.894898+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000919180/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:16:25.844861+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001350000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:16:25.84493+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0012ca000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:16:25.896194+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000919180/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:16:27.829971+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0012ca000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2024-05-30T15:16:27.830044+0800","logger":"etcd-client","caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-05-30T15:16:27.834926+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001350000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2024-05-30T15:16:27.835033+0800","logger":"etcd-client","caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-05-30T15:16:27.883971+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000919180/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":"2024-05-30T15:16:27.884041+0800","logger":"etcd-client","caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:14 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:16:31.846083+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0012ca000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:16:31.846567+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001350000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:16:31.897245+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000919180/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:15 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:17 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:18 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:16:37.8472+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0012ca000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:16:37.847921+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001350000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:16:37.899149+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000919180/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"warn","ts":1717053398.1149104,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000e4afc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1717053398.114981,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1717053398.1823962,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0039b0e00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1717053398.1824777,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:21 --:--:-- 0{"level":"warn","ts":1717053399.0314083,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00248e380/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1717053399.0314786,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:22 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:23 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:24 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:25 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:26 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:16:43.847966+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0012ca000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:16:43.849322+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001350000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:16:43.900091+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000919180/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:27 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:28 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 27 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 33 + synced_status='{ "error_msg": "[CDC:ErrPDEtcdAPIError]etcd api call error: context deadline exceeded", "error_code": "CDC:ErrPDEtcdAPIError" }' ++ echo '{' '"error_msg":' '"[CDC:ErrPDEtcdAPIError]etcd' api call error: context deadline 'exceeded",' '"error_code":' '"CDC:ErrPDEtcdAPIError"' '}' ++ jq -r .error_code + error_code=CDC:ErrPDEtcdAPIError + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tikv conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2170ca000013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:27732, start at 2024-05-30 15:17:16.579310057 +0800 CST m=+5.399798358 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:19:16.587 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:17:16.594 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:07:16.594 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2170ca000013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:27732, start at 2024-05-30 15:17:16.579310057 +0800 CST m=+5.399798358 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:19:16.587 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:17:16.594 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:07:16.594 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2170cb700014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:27794, start at 2024-05-30 15:17:16.663673116 +0800 CST m=+5.412835560 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:19:16.671 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:17:16.636 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:07:16.636 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.29038.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='450115257484967937 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115257484967937 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=450115257484967937 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Thu May 30 15:17:23 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2906829070.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:17:26 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bb681a62-014f-4986-9bc3-e052740cff2f {"id":"bb681a62-014f-4986-9bc3-e052740cff2f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc85c0590c4 bb681a62-014f-4986-9bc3-e052740cff2f /tidb/cdc/default/default/upstream/7374688313416206126 {"id":7374688313416206126,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bb681a62-014f-4986-9bc3-e052740cff2f {"id":"bb681a62-014f-4986-9bc3-e052740cff2f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc85c0590c4 bb681a62-014f-4986-9bc3-e052740cff2f /tidb/cdc/default/default/upstream/7374688313416206126 {"id":7374688313416206126,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bb681a62-014f-4986-9bc3-e052740cff2f {"id":"bb681a62-014f-4986-9bc3-e052740cff2f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc85c0590c4 bb681a62-014f-4986-9bc3-e052740cff2f /tidb/cdc/default/default/upstream/7374688313416206126 {"id":7374688313416206126,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=450115257484967937 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.29113.out cli changefeed create --start-ts=450115257484967937 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7374688313416206126,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T15:17:26.94148273+08:00","start_ts":450115257484967937,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115257484967937,"checkpoint_ts":450115257484967937,"checkpoint_time":"2024-05-30 15:17:21.944"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 + kill_tikv ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 27116 24.7 0.4 3805828 1730932 ? Sl 15:17 0:06 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20160 --status-addr 127.0.0.1:20181 --log-file /tmp/tidb_cdc_test/synced_status/tikv1.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv1 jenkins 27117 16.7 0.4 3778696 1662960 ? Sl 15:17 0:04 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20161 --status-addr 127.0.0.1:20182 --log-file /tmp/tidb_cdc_test/synced_status/tikv2.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv2 jenkins 27118 17.0 0.4 3773064 1677456 ? Sl 15:17 0:04 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20162 --status-addr 127.0.0.1:20183 --log-file /tmp/tidb_cdc_test/synced_status/tikv3.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv3 jenkins 27120 23.1 0.4 3795588 1719420 ? Sl 15:17 0:06 tikv-server --pd 127.0.0.1:2479 -A 127.0.0.1:21160 --status-addr 127.0.0.1:21180 --log-file /tmp/tidb_cdc_test/synced_status/tikv_down.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv_down' ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 2952 0 --:--:-- --:--:-- --:--:-- 2963 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-30 15:17:34.244","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"2024-05-30 15:17:28.895","now_ts":"2024-05-30 15:17:36.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:17:34.244","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-05-30' '15:17:28.895","now_ts":"2024-05-30' '15:17:36.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:17:34.244","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-05-30' '15:17:28.895","now_ts":"2024-05-30' '15:17:36.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 9240 0 --:--:-- --:--:-- --:--:-- 9151 100 723 100 723 0 0 9230 0 --:--:-- --:--:-- --:--:-- 9151 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-30 15:17:36.294","puller_resolved_ts":"2024-05-30 15:17:36.294","last_synced_ts":"2024-05-30 15:17:28.895","now_ts":"2024-05-30 15:19:46.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:17:36.294","puller_resolved_ts":"2024-05-30' '15:17:36.294","last_synced_ts":"2024-05-30' '15:17:28.895","now_ts":"2024-05-30' '15:19:46.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:17:36.294","puller_resolved_ts":"2024-05-30' '15:17:36.294","last_synced_ts":"2024-05-30' '15:17:28.895","now_ts":"2024-05-30' '15:19:46.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tidb conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f217b6cac0003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:30230, start at 2024-05-30 15:20:10.796155746 +0800 CST m=+5.267932843 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:22:10.803 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:20:10.795 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:10:10.795 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f217b6cac0003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:30230, start at 2024-05-30 15:20:10.796155746 +0800 CST m=+5.267932843 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:22:10.803 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:20:10.795 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:10:10.795 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f217b6d800013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:30315, start at 2024-05-30 15:20:10.875032126 +0800 CST m=+5.296968776 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:22:10.882 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:20:10.848 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:10:10.848 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.31581.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='450115303176667140 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115303176667140 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=450115303176667140 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Thu May 30 15:20:17 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.3161731619.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:20:20 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3a5676f0-7812-4d5f-9d67-39ba5877b0b1 {"id":"3a5676f0-7812-4d5f-9d67-39ba5877b0b1","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc85eb090c4 3a5676f0-7812-4d5f-9d67-39ba5877b0b1 /tidb/cdc/default/default/upstream/7374689059910389958 {"id":7374689059910389958,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3a5676f0-7812-4d5f-9d67-39ba5877b0b1 {"id":"3a5676f0-7812-4d5f-9d67-39ba5877b0b1","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc85eb090c4 3a5676f0-7812-4d5f-9d67-39ba5877b0b1 /tidb/cdc/default/default/upstream/7374689059910389958 {"id":7374689059910389958,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3a5676f0-7812-4d5f-9d67-39ba5877b0b1 {"id":"3a5676f0-7812-4d5f-9d67-39ba5877b0b1","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc85eb090c4 3a5676f0-7812-4d5f-9d67-39ba5877b0b1 /tidb/cdc/default/default/upstream/7374689059910389958 {"id":7374689059910389958,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=450115303176667140 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.31663.out cli changefeed create --start-ts=450115303176667140 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7374689059910389958,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T15:20:21.202901147+08:00","start_ts":450115303176667140,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115303176667140,"checkpoint_ts":450115303176667140,"checkpoint_time":"2024-05-30 15:20:16.244"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 + kill_tidb ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 30226 3.7 0.0 2778692 164348 ? Sl 15:20 0:00 tidb-server -P 4000 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1717053605517892894.toml --store tikv --path 127.0.0.1:2379 --status=10080 --log-file /tmp/tidb_cdc_test/synced_status/tidb.log jenkins 30230 11.5 0.0 2953080 189548 ? Sl 15:20 0:02 tidb-server -P 4001 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1717053605520961479.toml --store tikv --path 127.0.0.1:2379 --status=10081 --log-file /tmp/tidb_cdc_test/synced_status/tidb_other.log jenkins 30315 11.9 0.0 2714852 216784 ? Sl 15:20 0:02 tidb-server -P 3306 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1717053605571431073.toml --store tikv --path 127.0.0.1:2479 --status=20080 --log-file /tmp/tidb_cdc_test/synced_status/tidb_down.log' ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 3452 0 --:--:-- --:--:-- --:--:-- 3471 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-30 15:20:28.794","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"2024-05-30 15:20:23.194","now_ts":"2024-05-30 15:20:30.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:20:28.794","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-05-30' '15:20:23.194","now_ts":"2024-05-30' '15:20:30.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:20:28.794","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-05-30' '15:20:23.194","now_ts":"2024-05-30' '15:20:30.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2747 0 --:--:-- --:--:-- --:--:-- 2762 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-30 15:22:38.894","puller_resolved_ts":"2024-05-30 15:22:32.894","last_synced_ts":"2024-05-30 15:20:23.194","now_ts":"2024-05-30 15:22:40.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:22:38.894","puller_resolved_ts":"2024-05-30' '15:22:32.894","last_synced_ts":"2024-05-30' '15:20:23.194","now_ts":"2024-05-30' '15:22:40.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:22:38.894","puller_resolved_ts":"2024-05-30' '15:22:32.894","last_synced_ts":"2024-05-30' '15:20:23.194","now_ts":"2024-05-30' '15:22:40.000","info":"Data' syncing is 'finished"}' ++ jq -r .info + info='Data syncing is finished' + target_message='Data syncing is finished' + '[' 'Data syncing is finished' '!=' 'Data syncing is finished' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_failpoint conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21864c540014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:32749, start at 2024-05-30 15:23:08.995988803 +0800 CST m=+5.083449911 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:25:09.002 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:23:08.999 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:13:08.999 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21864c540014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:32749, start at 2024-05-30 15:23:08.995988803 +0800 CST m=+5.083449911 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:25:09.002 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:23:08.999 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:13:08.999 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21865b100014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-590-pd51w, pid:32841, start at 2024-05-30 15:23:09.932291768 +0800 CST m=+5.963123517 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:25:09.938 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:23:09.941 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:13:09.941 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status + export 'GO_FAILPOINTS=github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.34157.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='450115349393965057 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115349393965057 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=450115349393965057 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Thu May 30 15:23:13 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.3419534197.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:23:17 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b09dd3cb-c501-4784-9bf8-e4da84560fbd {"id":"b09dd3cb-c501-4784-9bf8-e4da84560fbd","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc861623bbf b09dd3cb-c501-4784-9bf8-e4da84560fbd /tidb/cdc/default/default/upstream/7374689828127029519 {"id":7374689828127029519,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b09dd3cb-c501-4784-9bf8-e4da84560fbd {"id":"b09dd3cb-c501-4784-9bf8-e4da84560fbd","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc861623bbf b09dd3cb-c501-4784-9bf8-e4da84560fbd /tidb/cdc/default/default/upstream/7374689828127029519 {"id":7374689828127029519,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b09dd3cb-c501-4784-9bf8-e4da84560fbd {"id":"b09dd3cb-c501-4784-9bf8-e4da84560fbd","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc861623bbf b09dd3cb-c501-4784-9bf8-e4da84560fbd /tidb/cdc/default/default/upstream/7374689828127029519 {"id":7374689828127029519,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=450115349393965057 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.34245.out cli changefeed create --start-ts=450115349393965057 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7374689828127029519,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T15:23:17.466829807+08:00","start_ts":450115349393965057,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115349393965057,"checkpoint_ts":450115349393965057,"checkpoint_time":"2024-05-30 15:23:12.549"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x + sleep 20 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 7574 0 --:--:-- --:--:-- --:--:-- 7610 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-30 15:23:12.549","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-05-30 15:23:38.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:23:12.549","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-30' '15:23:38.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:23:12.549","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-30' '15:23:38.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + export GO_FAILPOINTS= + GO_FAILPOINTS= + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + check_logs /tmp/tidb_cdc_test/synced_status ++ date + echo '[Thu May 30 15:23:51 CST 2024] <<<<<< run test case synced_status success! >>>>>>' [Thu May 30 15:23:51 CST 2024] <<<<<< run test case synced_status success! >>>>>> + stop_tidb_cluster \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-590/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // parallel [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] End of Pipeline Finished: SUCCESS