Started by user Jenkins Admin Obtained pipelines/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test.groovy from git https://github.com/PingCAP-QE/ci.git Loading library tipipeline@main Library tipipeline@main is cached. Copying from home. [Pipeline] Start of Pipeline [Pipeline] readJSON [Pipeline] readTrusted Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8jf5j Agent p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8jf5j is provisioned from template p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-t9qld-8vb6d --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "3c035b756ad2ec647fa3f70e4077fff58a93bc2d" jenkins/label: "p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-t9qld" name: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8jf5j" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - command: - "bin/pulsar" - "standalone" image: "apachepulsar/pulsar:2.10.4" imagePullPolicy: "IfNotPresent" name: "pulsar" resources: limits: memory: "6Gi" cpu: "6" requests: memory: "4Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8jf5j" - name: "JENKINS_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8jf5j" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8jf5j in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout The recommended git tool is: git No credentials specified Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 > git rev-list --no-walk e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] timeout Timeout set to expire in 1 hr 0 min [Pipeline] { [Pipeline] stage [Pipeline] { (Debug info) [Pipeline] sh + printenv PROW_JOB_ID=f6375c57-288c-4d36-bd8b-6f8b308ffae9 JENKINS_NODE_COOKIE=23b91221-2a59-47b3-aa25-3540c6dff015 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/ GOLANG_VERSION=1.21.10 HOSTNAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8jf5j HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Debug info BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test","buildid":"1796069089735610368","prowjobid":"f6375c57-288c-4d36-bd8b-6f8b308ffae9","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct _=/usr/bin/printenv POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#462 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089735610368 GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=tests SHLVL=3 HOME=/home/jenkins POD_LABEL=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-t9qld GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-ca6cce641717f42bbc3d648d43f3161c6159c21ce0cc69f0c87db19704c358e7 NODE_LABELS=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8jf5j p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-t9qld GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8jf5j GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=462 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz + echo ------------------------- ------------------------- + go env GO111MODULE='' GOARCH='amd64' GOBIN='' GOCACHE='/home/jenkins/.cache/go-build' GOENV='/home/jenkins/.config/go/env' GOEXE='' GOEXPERIMENT='' GOFLAGS='' GOHOSTARCH='amd64' GOHOSTOS='linux' GOINSECURE='' GOMODCACHE='/go/pkg/mod' GONOPROXY='' GONOSUMDB='' GOOS='linux' GOPATH='/go' GOPRIVATE='' GOPROXY='http://goproxy.apps.svc,https://proxy.golang.org,direct' GOROOT='/usr/local/go' GOSUMDB='sum.golang.org' GOTMPDIR='' GOTOOLCHAIN='auto' GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64' GOVCS='' GOVERSION='go1.21.10' GCCGO='gccgo' GOAMD64='v1' AR='ar' CC='gcc' CXX='g++' CGO_ENABLED='1' GOMOD='/dev/null' GOWORK='' CGO_CFLAGS='-O2 -g' CGO_CPPFLAGS='' CGO_CXXFLAGS='-O2 -g' CGO_FFLAGS='-O2 -g' CGO_LDFLAGS='-O2 -g' PKG_CONFIG='pkg-config' GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2214935883=/tmp/go-build -gno-record-gcc-switches' + echo ------------------------- ------------------------- + echo 'debug command: kubectl -n jenkins-tiflow exec -ti p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8jf5j bash' debug command: kubectl -n jenkins-tiflow exec -ti p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8jf5j bash [Pipeline] container [Pipeline] { [Pipeline] sh + dig github.com ; <<>> DiG 9.18.16 <<>> github.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9411 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ; COOKIE: 203df01044042649 (echoed) ;; QUESTION SECTION: ;github.com. IN A ;; ANSWER SECTION: github.com. 18 IN A 20.205.243.166 ;; Query time: 0 msec ;; SERVER: 169.254.25.10#53(169.254.25.10) (UDP) ;; WHEN: Thu May 30 06:40:55 UTC 2024 ;; MSG SIZE rcvd: 77 [Pipeline] script [Pipeline] { [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Check diff files) [Pipeline] container [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $token [Pipeline] { [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/11200/files?page=1&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/11200/files?page=1&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/11200/files?page=2&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/11200/files?page=2&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] } [Pipeline] // withCredentials [Pipeline] echo pr_diff_files: [cdc/kv/shared_client.go, go.mod, go.sum, pkg/spanz/hash_map.go, pkg/util/seahash/LICENSE.md, pkg/util/seahash/seahash.go, pkg/util/seahash/seahash_test.go, scripts/check-copyright.sh] [Pipeline] echo diff file not matched: cdc/kv/shared_client.go [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Checkout) [Pipeline] timeout Timeout set to expire in 10 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] cache Cache restored successfully (git/pingcap/tiflow/rev-26b901c-8f92142) 208203776 bytes in 1.23 secs (168638749 bytes/sec) [Pipeline] { [Pipeline] retry [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] sh git version 2.36.6 Reinitialized existing Git repository in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/.git/ .git HEAD is now at 8f92142d7 fix POST git-upload-pack (686 bytes) From https://github.com/pingcap/tiflow = [up to date] release-7.5 -> origin/release-7.5 = [up to date] refs/pull/11200/head -> origin/pr/11200/head Previous HEAD position was 8f92142d7 fix HEAD is now at 26b901cb5 dm: upgrade go-mysql library (#11043) (#11189) ๐Ÿšง Checkouting to base SHA:26b901cb5b23f8c42b05e5a00514b52bef2efe24... HEAD is now at 26b901cb5 dm: upgrade go-mysql library (#11043) (#11189) โœ… Checked. ๐ŸŽ‰ ๐Ÿงพ HEAD info: 26b901cb5b23f8c42b05e5a00514b52bef2efe24 26b901cb5 dm: upgrade go-mysql library (#11043) (#11189) bf93d7d63 dm: fix update dml loss if binary column is primary key (#10685) (#10691) 39a17af6e dep(*): move diff package to the tiflow and get rid of the tidb-tools (#11164) (#11168) ๐Ÿšง Pre-merge heads of pull requests to base SHA: 26b901cb5b23f8c42b05e5a00514b52bef2efe24 ... Updating 26b901cb5..8f92142d7 Fast-forward cdc/kv/shared_client.go | 2 +- go.mod | 10 --- go.sum | 34 --------- pkg/spanz/hash_map.go | 2 +- pkg/util/seahash/LICENSE.md | 9 +++ pkg/util/seahash/seahash.go | 159 +++++++++++++++++++++++++++++++++++++++ pkg/util/seahash/seahash_test.go | 75 ++++++++++++++++++ scripts/check-copyright.sh | 2 +- 8 files changed, 246 insertions(+), 47 deletions(-) create mode 100644 pkg/util/seahash/LICENSE.md create mode 100644 pkg/util/seahash/seahash.go create mode 100644 pkg/util/seahash/seahash_test.go ๐Ÿงพ Pre-merged result: 8f92142d7874bfad2a8db6ca100604c383fca9df 8f92142d7 fix 645f6fc14 This is an automated cherry-pick of #11193 26b901cb5 dm: upgrade go-mysql library (#11043) (#11189) โœ… Pre merged ๐ŸŽ‰ โœ… ~~~~~All done.~~~~~~ [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // retry [Pipeline] } Cache not saved (git/pingcap/tiflow/rev-26b901c-8f92142 already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (prepare) [Pipeline] timeout Timeout set to expire in 20 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/third_party_download [Pipeline] { [Pipeline] retry [Pipeline] { [Pipeline] sh + cd ../tiflow + ./scripts/download-integration-test-binaries.sh release-7.5 Download binaries... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 41 100 41 0 0 790 0 --:--:-- --:--:-- --:--:-- 803 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 41 100 41 0 0 1319 0 --:--:-- --:--:-- --:--:-- 1366 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 41 100 41 0 0 1583 0 --:--:-- --:--:-- --:--:-- 1640 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 41 100 41 0 0 2545 0 --:--:-- --:--:-- --:--:-- 2733 >>> download tidb-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tidb/39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef/centos7/tidb-server.tar.gz 2024-05-30 14:41:27 URL:http://fileserver.pingcap.net/download/builds/pingcap/tidb/39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef/centos7/tidb-server.tar.gz [512424861/512424861] -> "tmp/tidb-server.tar.gz" [1] >>> download pd-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/pd/634e05a87ee0459846e48e8acea3f6bc5f6973a7/centos7/pd-server.tar.gz 2024-05-30 14:41:47 URL:http://fileserver.pingcap.net/download/builds/pingcap/pd/634e05a87ee0459846e48e8acea3f6bc5f6973a7/centos7/pd-server.tar.gz [173743994/173743994] -> "tmp/pd-server.tar.gz" [1] >>> download tikv-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tikv/1a2c88dcc22c62e1f138d440956214ce367e838b/centos7/tikv-server.tar.gz 2024-05-30 14:42:31 URL:http://fileserver.pingcap.net/download/builds/pingcap/tikv/1a2c88dcc22c62e1f138d440956214ce367e838b/centos7/tikv-server.tar.gz [817702020/817702020] -> "tmp/tikv-server.tar.gz" [1] >>> download tiflash.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tiflash/release-7.5/db0862135d4a94904069a37ec50c69cfe6099952/centos7/tiflash.tar.gz 2024-05-30 14:42:55 URL:http://fileserver.pingcap.net/download/builds/pingcap/tiflash/release-7.5/db0862135d4a94904069a37ec50c69cfe6099952/centos7/tiflash.tar.gz [263371625/263371625] -> "tmp/tiflash.tar.gz" [1] >>> download minio.tar.gz from http://fileserver.pingcap.net/download/minio.tar.gz 2024-05-30 14:42:59 URL:http://fileserver.pingcap.net/download/minio.tar.gz [17718777/17718777] -> "tmp/minio.tar.gz" [1] >>> download go-ycsb from http://fileserver.pingcap.net/download/builds/pingcap/go-ycsb/test-br/go-ycsb 2024-05-30 14:43:00 URL:http://fileserver.pingcap.net/download/builds/pingcap/go-ycsb/test-br/go-ycsb [45975512/45975512] -> "third_bin/go-ycsb" [1] >>> download jq from http://fileserver.pingcap.net/download/builds/pingcap/test/jq-1.6/jq-linux64 2024-05-30 14:43:01 URL:http://fileserver.pingcap.net/download/builds/pingcap/test/jq-1.6/jq-linux64 [3953824/3953824] -> "third_bin/jq" [1] >>> download etcd.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/etcd-v3.4.7-linux-amd64.tar.gz 2024-05-30 14:43:01 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/etcd-v3.4.7-linux-amd64.tar.gz [17310840/17310840] -> "tmp/etcd.tar.gz" [1] >>> download sync_diff_inspector.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/sync_diff_inspector_hash-79f1fd1e_linux-amd64.tar.gz 2024-05-30 14:43:04 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/sync_diff_inspector_hash-79f1fd1e_linux-amd64.tar.gz [55896256/55896256] -> "tmp/sync_diff_inspector.tar.gz" [1] >>> download schema-registry.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/schema-registry.tar.gz 2024-05-30 14:43:11 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/schema-registry.tar.gz [278386006/278386006] -> "tmp/schema-registry.tar.gz" [1] Download SUCCESS + ls -alh ./bin total 1.6G drwxr-sr-x. 6 jenkins jenkins 4.0K May 30 14:43 . drwxr-sr-x. 19 jenkins jenkins 4.0K May 30 14:43 .. drwxr-sr-x. 2 jenkins jenkins 4.0K May 19 2023 bin drwxr-sr-x. 4 jenkins jenkins 4.0K May 10 2023 etc -rwxr-xr-x. 1 jenkins jenkins 17M Apr 2 2020 etcdctl -rwxr-xr-x. 1 jenkins jenkins 44M May 30 14:43 go-ycsb -rwxr-xr-x. 1 jenkins jenkins 3.8M May 30 14:43 jq drwxr-sr-x. 3 jenkins jenkins 4.0K May 10 2023 lib lrwxrwxrwx. 1 jenkins jenkins 13 May 27 16:52 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x. 1 jenkins jenkins 1016K Nov 7 2023 libc++.so.1.0 lrwxrwxrwx. 1 jenkins jenkins 16 May 27 16:52 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x. 1 jenkins jenkins 358K Nov 7 2023 libc++abi.so.1.0 lrwxrwxrwx. 1 jenkins jenkins 13 May 27 16:52 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx. 1 jenkins jenkins 15 May 27 16:52 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x. 1 jenkins jenkins 2.6M May 27 16:18 libgmssl.so.3.0 -rwxr-xr-x. 1 jenkins jenkins 88M May 27 16:44 libtiflash_proxy.so -rwxr-xr-x. 1 jenkins jenkins 50M Jul 29 2020 minio -rwxr-xr-x. 1 jenkins jenkins 33M May 28 20:27 pd-api-bench -rwxr-xr-x. 1 jenkins jenkins 43M May 28 20:27 pd-ctl -rwxr-xr-x. 1 jenkins jenkins 30M May 28 20:27 pd-heartbeat-bench -rwxr-xr-x. 1 jenkins jenkins 29M May 28 20:27 pd-recover -rwxr-xr-x. 1 jenkins jenkins 103M May 28 20:26 pd-server -rwxr-xr-x. 1 jenkins jenkins 25M May 28 20:27 pd-tso-bench -rwxr-xr-x. 1 jenkins jenkins 29M May 28 20:27 regions-dump drwxr-sr-x. 4 jenkins jenkins 4.0K May 10 2023 share -rwxr-xr-x. 1 jenkins jenkins 29M May 28 20:27 stores-dump -rwxr-xr-x. 1 jenkins jenkins 133M May 6 16:40 sync_diff_inspector -rwxr-xr-x. 1 jenkins jenkins 199M May 29 23:11 tidb-server -rwxr-xr-x. 1 jenkins jenkins 377M May 27 16:52 tiflash -rwxr-xr-x. 1 jenkins jenkins 357M May 28 18:52 tikv-server + make check_third_party_binary /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/bin/tidb-server /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/bin/tikv-server /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/bin/pd-server /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/bin/tiflash /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/bin/pd-ctl /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/bin/sync_diff_inspector /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/bin/go-ycsb /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/bin/etcdctl /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/bin/jq /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/bin/minio /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/bin/bin/schema-registry-start + cd - /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/third_party_download + mkdir -p bin + mv ../tiflow/bin/bin ../tiflow/bin/etc ../tiflow/bin/etcdctl ../tiflow/bin/go-ycsb ../tiflow/bin/jq ../tiflow/bin/lib ../tiflow/bin/libc++.so.1 ../tiflow/bin/libc++.so.1.0 ../tiflow/bin/libc++abi.so.1 ../tiflow/bin/libc++abi.so.1.0 ../tiflow/bin/libgmssl.so ../tiflow/bin/libgmssl.so.3 ../tiflow/bin/libgmssl.so.3.0 ../tiflow/bin/libtiflash_proxy.so ../tiflow/bin/minio ../tiflow/bin/pd-api-bench ../tiflow/bin/pd-ctl ../tiflow/bin/pd-heartbeat-bench ../tiflow/bin/pd-recover ../tiflow/bin/pd-server ../tiflow/bin/pd-tso-bench ../tiflow/bin/regions-dump ../tiflow/bin/share ../tiflow/bin/stores-dump ../tiflow/bin/sync_diff_inspector ../tiflow/bin/tidb-server ../tiflow/bin/tiflash ../tiflow/bin/tikv-server ./bin/ + ls -alh ./bin total 1.6G drwxr-sr-x. 6 jenkins jenkins 4.0K May 30 14:43 . drwxr-sr-x. 3 jenkins jenkins 4.0K May 30 14:43 .. drwxr-sr-x. 2 jenkins jenkins 4.0K May 19 2023 bin drwxr-sr-x. 4 jenkins jenkins 4.0K May 10 2023 etc -rwxr-xr-x. 1 jenkins jenkins 17M Apr 2 2020 etcdctl -rwxr-xr-x. 1 jenkins jenkins 44M May 30 14:43 go-ycsb -rwxr-xr-x. 1 jenkins jenkins 3.8M May 30 14:43 jq drwxr-sr-x. 3 jenkins jenkins 4.0K May 10 2023 lib lrwxrwxrwx. 1 jenkins jenkins 13 May 27 16:52 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x. 1 jenkins jenkins 1016K Nov 7 2023 libc++.so.1.0 lrwxrwxrwx. 1 jenkins jenkins 16 May 27 16:52 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x. 1 jenkins jenkins 358K Nov 7 2023 libc++abi.so.1.0 lrwxrwxrwx. 1 jenkins jenkins 13 May 27 16:52 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx. 1 jenkins jenkins 15 May 27 16:52 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x. 1 jenkins jenkins 2.6M May 27 16:18 libgmssl.so.3.0 -rwxr-xr-x. 1 jenkins jenkins 88M May 27 16:44 libtiflash_proxy.so -rwxr-xr-x. 1 jenkins jenkins 50M Jul 29 2020 minio -rwxr-xr-x. 1 jenkins jenkins 33M May 28 20:27 pd-api-bench -rwxr-xr-x. 1 jenkins jenkins 43M May 28 20:27 pd-ctl -rwxr-xr-x. 1 jenkins jenkins 30M May 28 20:27 pd-heartbeat-bench -rwxr-xr-x. 1 jenkins jenkins 29M May 28 20:27 pd-recover -rwxr-xr-x. 1 jenkins jenkins 103M May 28 20:26 pd-server -rwxr-xr-x. 1 jenkins jenkins 25M May 28 20:27 pd-tso-bench -rwxr-xr-x. 1 jenkins jenkins 29M May 28 20:27 regions-dump drwxr-sr-x. 4 jenkins jenkins 4.0K May 10 2023 share -rwxr-xr-x. 1 jenkins jenkins 29M May 28 20:27 stores-dump -rwxr-xr-x. 1 jenkins jenkins 133M May 6 16:40 sync_diff_inspector -rwxr-xr-x. 1 jenkins jenkins 199M May 29 23:11 tidb-server -rwxr-xr-x. 1 jenkins jenkins 377M May 27 16:52 tiflash -rwxr-xr-x. 1 jenkins jenkins 357M May 28 18:52 tikv-server + ./bin/tidb-server -V Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore + ./bin/pd-server -V Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 + ./bin/tikv-server -V TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + ./bin/tiflash --version TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure + ./bin/sync_diff_inspector --version App Name: sync_diff_inspector v2.0 Release Version: v7.1.5-2-geae7b5d Git Commit Hash: eae7b5d149e30a02a04ec6116e08b0a8f07fe4aa Git Branch: disable-cgo-for-sync-diff UTC Build Time: 2024-05-06 08:40:40 Go Version: go1.22.1 [Pipeline] } [Pipeline] // retry [Pipeline] } [Pipeline] // dir [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] cache Cache not restored (no such key found) [Pipeline] { [Pipeline] sh + ls -alh ./bin total 8.0K drwxr-sr-x. 2 jenkins jenkins 4.0K May 30 14:43 . drwxr-sr-x. 19 jenkins jenkins 4.0K May 30 14:43 .. + '[' -f ./bin/cdc ']' + make cdc CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-58-g8f92142d7" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-30 06:43:15" -X "github.com/pingcap/tiflow/pkg/version.GitHash=8f92142d7874bfad2a8db6ca100604c383fca9df" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.10 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-58-g8f92142d7"' -o bin/cdc ./cmd/cdc/main.go go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20240527072219-a97e6464c01d go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading github.com/pingcap/failpoint v0.0.0-20240412033321-fd0796e60f86 go: downloading github.com/fatih/color v1.16.0 go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/spf13/pflag v1.0.5 go: downloading go.uber.org/zap v1.27.0 go: downloading golang.org/x/net v0.25.0 go: downloading github.com/BurntSushi/toml v1.3.2 go: downloading github.com/IBM/sarama v1.41.2 go: downloading github.com/spf13/cobra v1.8.0 go: downloading github.com/pingcap/kvproto v0.0.0-20240522024016-df42997c2c57 go: downloading github.com/xdg/scram v1.0.5 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading google.golang.org/grpc v1.64.0 go: downloading github.com/tikv/pd/client v0.0.0-20240210135946-3488a653ddd9 go: downloading github.com/gin-gonic/gin v1.9.1 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/prometheus/client_golang v1.19.1 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20240510070134-e7f530c64790 go: downloading golang.org/x/sync v0.7.0 go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20240527072219-a97e6464c01d go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/google/uuid v1.6.0 go: downloading go.etcd.io/etcd/server/v3 v3.5.12 go: downloading golang.org/x/time v0.5.0 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading github.com/swaggo/files v0.0.0-20210815190702-a29dd2bc99b2 go: downloading github.com/swaggo/gin-swagger v1.2.0 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/aws/aws-sdk-go-v2 v1.19.1 go: downloading github.com/apache/pulsar-client-go v0.11.0 go: downloading github.com/cockroachdb/pebble v0.0.0-20220415182917-06c9d3be25b3 go: downloading go.uber.org/atomic v1.11.0 go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/soheilhy/cmux v0.1.5 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.1 go: downloading github.com/KimMachineGun/automemlimit v0.2.4 go: downloading github.com/aws/aws-sdk-go v1.44.259 go: downloading github.com/shirou/gopsutil/v3 v3.24.4 go: downloading github.com/tinylib/msgp v1.1.6 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading github.com/stretchr/testify v1.9.0 go: downloading golang.org/x/sys v0.20.0 go: downloading github.com/DATA-DOG/go-sqlmock v1.5.0 go: downloading github.com/imdario/mergo v0.3.16 go: downloading github.com/cenkalti/backoff/v4 v4.2.1 go: downloading cloud.google.com/go v0.112.2 go: downloading github.com/goccy/go-json v0.10.2 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading github.com/benbjohnson/clock v1.3.5 go: downloading github.com/r3labs/diff v1.1.0 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading github.com/google/btree v1.1.2 go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading github.com/uber-go/atomic v1.4.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/swaggo/swag v1.16.3 go: downloading github.com/klauspost/compress v1.17.8 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/xdg/stringprep v1.0.3 go: downloading golang.org/x/crypto v0.23.0 go: downloading github.com/golang/mock v1.6.0 go: downloading github.com/containerd/cgroups v1.0.4 go: downloading github.com/gin-contrib/sse v0.1.0 go: downloading github.com/philhofer/fwd v1.1.1 go: downloading github.com/gavv/monotime v0.0.0-20190418164738-30dba4353424 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/hashicorp/golang-lru v0.5.1 go: downloading github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 go: downloading golang.org/x/oauth2 v0.18.0 go: downloading github.com/jcmturner/gokrb5/v8 v8.4.4 go: downloading github.com/segmentio/kafka-go v0.4.41-0.20230526171612-f057b1d369cd go: downloading google.golang.org/protobuf v1.34.1 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/pingcap/check v0.0.0-20211026125417-57bd13f7b5f0 go: downloading github.com/go-playground/validator/v10 v10.14.0 go: downloading github.com/go-mysql-org/go-mysql v1.7.1-0.20240507075657-2bd4573edde2 go: downloading github.com/ugorji/go/codec v1.2.11 go: downloading github.com/pelletier/go-toml/v2 v2.0.8 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/prometheus/common v0.53.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/prometheus/procfs v0.15.0 go: downloading go.etcd.io/etcd/pkg/v3 v3.5.12 go: downloading github.com/aws/smithy-go v1.13.5 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/golang/protobuf v1.5.4 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/eapache/go-resiliency v1.4.0 go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 go: downloading github.com/eapache/queue v1.1.0 go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/jcmturner/gofork v1.7.6 go: downloading github.com/bits-and-blooms/bitset v1.4.0 go: downloading github.com/linkedin/goavro/v2 v2.11.1 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading github.com/stretchr/objx v0.5.2 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading google.golang.org/api v0.170.0 go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 go: downloading go.etcd.io/bbolt v1.3.9 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.22.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading go.opentelemetry.io/otel/sdk v1.22.0 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading golang.org/x/text v0.15.0 go: downloading github.com/KyleBanks/depth v1.2.1 go: downloading github.com/go-openapi/spec v0.21.0 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/godbus/dbus/v5 v5.0.4 go: downloading github.com/opencontainers/runtime-spec v1.0.2 go: downloading github.com/cilium/ebpf v0.4.0 go: downloading github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 go: downloading github.com/cakturk/go-netstat v0.0.0-20200220111822-e5b49efee7a5 go: downloading github.com/AthenZ/athenz v1.10.39 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240521202816-d264139d666e go: downloading github.com/jcmturner/dnsutils/v2 v2.0.0 go: downloading github.com/hashicorp/go-uuid v1.0.3 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.1 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/pierrec/lz4 v2.6.1+incompatible go: downloading github.com/gabriel-vasile/mimetype v1.4.2 go: downloading github.com/go-playground/universal-translator v0.18.1 go: downloading github.com/leodido/go-urn v1.2.4 go: downloading github.com/Masterminds/semver v1.5.0 go: downloading github.com/siddontang/go-log v0.0.0-20180807004314-8d05993dda07 go: downloading github.com/siddontang/go v0.0.0-20180604090527-bdc77568d726 go: downloading github.com/xdg-go/scram v1.1.2 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/hashicorp/errwrap v1.0.0 go: downloading github.com/spaolacci/murmur3 v1.1.0 go: downloading golang.org/x/mod v0.17.0 go: downloading github.com/golang/snappy v0.0.4 go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading github.com/gorilla/websocket v1.5.1 go: downloading go.etcd.io/etcd/raft/v3 v3.5.12 go: downloading github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 go: downloading github.com/jonboulle/clockwork v0.4.0 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading go.opentelemetry.io/proto/otlp v1.1.0 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/go-openapi/jsonpointer v0.21.0 go: downloading github.com/go-openapi/jsonreference v0.21.0 go: downloading github.com/go-openapi/swag v0.23.0 go: downloading github.com/aws/aws-sdk-go-v2/config v1.18.30 go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.13.29 go: downloading github.com/aws/aws-sdk-go-v2/service/glue v1.58.1 go: downloading github.com/jarcoal/httpmock v1.2.0 go: downloading github.com/mailru/easyjson v0.7.7 go: downloading github.com/shopspring/decimal v1.3.0 go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible go: downloading github.com/99designs/keyring v1.2.1 go: downloading github.com/jcmturner/aescts/v2 v2.0.0 go: downloading github.com/jcmturner/rpc/v2 v2.0.3 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.6 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/pingcap/tipb v0.0.0-20240507090649-2bf6bb0cb996 go: downloading github.com/go-logr/logr v1.4.1 go: downloading github.com/blacktear23/go-proxyprotocol v1.0.6 go: downloading github.com/gorilla/mux v1.8.0 go: downloading github.com/pingcap/fn v1.0.0 go: downloading github.com/golang-jwt/jwt/v4 v4.5.0 go: downloading go.etcd.io/etcd/client/v2 v2.305.12 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/tiancaiamao/appdash v0.0.0-20181126055449-889f96f722a2 go: downloading github.com/sourcegraph/appdash-data v0.0.0-20151005221446-73f23eafcf67 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.0 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/go-playground/locales v0.14.1 go: downloading github.com/xdg-go/pbkdf2 v1.0.0 go: downloading github.com/xdg-go/stringprep v1.0.4 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/dvsekhvalnov/jose2go v1.5.0 go: downloading github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c go: downloading github.com/mtibben/percent v0.2.1 go: downloading golang.org/x/term v0.20.0 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 go: downloading github.com/danjacques/gofslock v0.0.0-20220131014315-6e321f4509c8 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.6 go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.3.37 go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.12.14 go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.14 go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.20.1 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.1 go: downloading github.com/iancoleman/strcase v0.2.0 go: downloading github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.12.0 go: downloading github.com/josharian/intern v1.0.0 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/pingcap/badger v1.5.1-0.20230103063557-828f39b09b6d go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/vbauerster/mpb/v7 v7.5.3 go: downloading github.com/sasha-s/go-deadlock v0.3.1 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.36 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/mattn/go-runewidth v0.0.15 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.30 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading k8s.io/api v0.28.6 go: downloading github.com/emirpasic/gods v1.18.1 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/petermattis/goid v0.0.0-20231207134359-e60b3f734c67 go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.30 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/golang-jwt/jwt/v5 v5.2.0 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/ardielle/ardielle-go v1.5.2 go: downloading k8s.io/apimachinery v0.28.6 go: downloading github.com/google/gofuzz v1.2.0 go: downloading k8s.io/klog/v2 v2.120.1 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b go: downloading github.com/jmespath/go-jmespath v0.4.0 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading github.com/google/s2a-go v0.1.7 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading github.com/felixge/httpsnoop v1.0.4 + '[' -f ./bin/cdc_pulsar_consumer ']' + make pulsar_consumer CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-58-g8f92142d7" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-30 06:46:13" -X "github.com/pingcap/tiflow/pkg/version.GitHash=8f92142d7874bfad2a8db6ca100604c383fca9df" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.10 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-58-g8f92142d7"' -o bin/cdc_pulsar_consumer ./cmd/pulsar-consumer/main.go + '[' -f ./bin/cdc.test ']' + make integration_test_build cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/failpoint-ctl github.com/pingcap/failpoint/failpoint-ctl go: downloading github.com/pingcap/failpoint v0.0.0-20210316064728-7acb0f0a3dfd go: downloading github.com/sergi/go-diff v1.1.0 CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-58-g8f92142d7" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-30 06:46:21" -X "github.com/pingcap/tiflow/pkg/version.GitHash=8f92142d7874bfad2a8db6ca100604c383fca9df" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.10 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-58-g8f92142d7"' -o bin/cdc_storage_consumer ./cmd/storage-consumer/main.go CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-58-g8f92142d7" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-30 06:46:21" -X "github.com/pingcap/tiflow/pkg/version.GitHash=8f92142d7874bfad2a8db6ca100604c383fca9df" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.10 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-58-g8f92142d7"' -o bin/cdc_kafka_consumer ./cmd/kafka-consumer/main.go CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-58-g8f92142d7" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-30 06:46:21" -X "github.com/pingcap/tiflow/pkg/version.GitHash=8f92142d7874bfad2a8db6ca100604c383fca9df" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.10 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-58-g8f92142d7"' -o bin/cdc_pulsar_consumer ./cmd/pulsar-consumer/main.go $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl enable >/dev/null) go: downloading github.com/DataDog/zstd v1.5.5 go: downloading gorm.io/gorm v1.24.5 go: downloading github.com/PingCAP-QE/go-sqlsmith v0.0.0-20231213065948-336e064b488d go: downloading github.com/chzyer/readline v1.5.1 go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 go: downloading github.com/deepmap/oapi-codegen v1.9.0 go: downloading github.com/glebarez/sqlite v1.7.0 go: downloading github.com/gogo/gateway v1.1.0 go: downloading github.com/getkin/kin-openapi v0.80.0 go: downloading gorm.io/driver/mysql v1.3.3 go: downloading github.com/shurcooL/httpgzip v0.0.0-20190720172056-320755c1c1b0 go: downloading github.com/glebarez/go-sqlite v1.21.2 go: downloading github.com/syndtr/goleveldb v1.0.1-0.20210305035536-64b5b1c73954 go: downloading github.com/go-ozzo/ozzo-validation/v4 v4.3.0 go: downloading github.com/mattn/go-shellwords v1.0.12 go: downloading github.com/edwingeng/deque v0.0.0-20191220032131-8596380dee17 go: downloading go.uber.org/dig v1.13.0 go: downloading github.com/VividCortex/mysqlerr v1.0.0 go: downloading go.uber.org/ratelimit v0.2.0 go: downloading go.uber.org/goleak v1.3.0 go: downloading github.com/ngaut/log v0.0.0-20210830112240-0124ec040aeb go: downloading github.com/integralist/go-findroot v0.0.0-20160518114804-ac90681525dc go: downloading github.com/bradleyjkemp/grpc-tools v0.2.5 go: downloading upper.io/db.v3 v3.7.1+incompatible go: downloading github.com/jmoiron/sqlx v1.3.3 go: downloading github.com/jinzhu/now v1.1.5 go: downloading github.com/jinzhu/inflection v1.0.0 go: downloading github.com/andres-erbsen/clock v0.0.0-20160526145045-9e14626cd129 go: downloading modernc.org/libc v1.50.4 go: downloading modernc.org/sqlite v1.29.8 go: downloading github.com/improbable-eng/grpc-web v0.12.0 go: downloading github.com/ghodss/yaml v1.0.0 go: downloading github.com/rs/cors v1.7.0 go: downloading github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f go: downloading modernc.org/memory v1.8.0 go: downloading modernc.org/mathutil v1.6.0 CGO_ENABLED=1 GO111MODULE=on go test -p 3 --race --tags=intest -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-58-g8f92142d7" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-30 06:46:21" -X "github.com/pingcap/tiflow/pkg/version.GitHash=8f92142d7874bfad2a8db6ca100604c383fca9df" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.10 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-58-g8f92142d7"' -c -cover -covermode=atomic \ -coverpkg=github.com/pingcap/tiflow/... \ -o bin/cdc.test github.com/pingcap/tiflow/cmd/cdc \ || { $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null); exit 1; } CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-58-g8f92142d7" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-30 06:46:21" -X "github.com/pingcap/tiflow/pkg/version.GitHash=8f92142d7874bfad2a8db6ca100604c383fca9df" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.10 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-58-g8f92142d7"' -o bin/cdc ./cmd/cdc/main.go \ || { $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null); exit 1; } $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null) + ls -alh ./bin total 1.1G drwxr-sr-x. 2 jenkins jenkins 4.0K May 30 14:51 . drwxr-sr-x. 19 jenkins jenkins 4.0K May 30 14:43 .. -rwxr-xr-x. 1 jenkins jenkins 208M May 30 14:51 cdc -rwxr-xr-x. 1 jenkins jenkins 339M May 30 14:50 cdc.test -rwxr-xr-x. 1 jenkins jenkins 178M May 30 14:46 cdc_kafka_consumer -rwxr-xr-x. 1 jenkins jenkins 177M May 30 14:46 cdc_pulsar_consumer -rwxr-xr-x. 1 jenkins jenkins 177M May 30 14:46 cdc_storage_consumer + ./bin/cdc version Release Version: v7.5.1-58-g8f92142d7 Git Commit Hash: 8f92142d7874bfad2a8db6ca100604c383fca9df Git Branch: HEAD UTC Build Time: 2024-05-30 06:46:21 Go Version: go version go1.21.10 linux/amd64 Failpoint Build: true [Pipeline] } Cache saved successfully (binary/pingcap/tiflow/cdc-integration-pulsar-test/rev-26b901c-8f92142) 1129025024 bytes in 17.79 secs (63474296 bytes/sec) [Pipeline] // cache [Pipeline] cache Cache not restored (no such key found) [Pipeline] { [Pipeline] sh + cp -r ../third_party_download/bin/bin ../third_party_download/bin/etc ../third_party_download/bin/etcdctl ../third_party_download/bin/go-ycsb ../third_party_download/bin/jq ../third_party_download/bin/lib ../third_party_download/bin/libc++.so.1 ../third_party_download/bin/libc++.so.1.0 ../third_party_download/bin/libc++abi.so.1 ../third_party_download/bin/libc++abi.so.1.0 ../third_party_download/bin/libgmssl.so ../third_party_download/bin/libgmssl.so.3 ../third_party_download/bin/libgmssl.so.3.0 ../third_party_download/bin/libtiflash_proxy.so ../third_party_download/bin/minio ../third_party_download/bin/pd-api-bench ../third_party_download/bin/pd-ctl ../third_party_download/bin/pd-heartbeat-bench ../third_party_download/bin/pd-recover ../third_party_download/bin/pd-server ../third_party_download/bin/pd-tso-bench ../third_party_download/bin/regions-dump ../third_party_download/bin/share ../third_party_download/bin/stores-dump ../third_party_download/bin/sync_diff_inspector ../third_party_download/bin/tidb-server ../third_party_download/bin/tiflash ../third_party_download/bin/tikv-server ./bin/ + ls -alh ./bin total 2.7G drwxr-sr-x. 6 jenkins jenkins 4.0K May 30 14:51 . drwxr-sr-x. 19 jenkins jenkins 4.0K May 30 14:43 .. drwxr-sr-x. 2 jenkins jenkins 4.0K May 30 14:51 bin -rwxr-xr-x. 1 jenkins jenkins 208M May 30 14:51 cdc -rwxr-xr-x. 1 jenkins jenkins 339M May 30 14:50 cdc.test -rwxr-xr-x. 1 jenkins jenkins 178M May 30 14:46 cdc_kafka_consumer -rwxr-xr-x. 1 jenkins jenkins 177M May 30 14:46 cdc_pulsar_consumer -rwxr-xr-x. 1 jenkins jenkins 177M May 30 14:46 cdc_storage_consumer drwxr-sr-x. 4 jenkins jenkins 4.0K May 30 14:51 etc -rwxr-xr-x. 1 jenkins jenkins 17M May 30 14:51 etcdctl -rwxr-xr-x. 1 jenkins jenkins 44M May 30 14:51 go-ycsb -rwxr-xr-x. 1 jenkins jenkins 3.8M May 30 14:51 jq drwxr-sr-x. 3 jenkins jenkins 4.0K May 30 14:51 lib lrwxrwxrwx. 1 jenkins jenkins 13 May 30 14:51 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x. 1 jenkins jenkins 1016K May 30 14:51 libc++.so.1.0 lrwxrwxrwx. 1 jenkins jenkins 16 May 30 14:51 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x. 1 jenkins jenkins 358K May 30 14:51 libc++abi.so.1.0 lrwxrwxrwx. 1 jenkins jenkins 13 May 30 14:51 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx. 1 jenkins jenkins 15 May 30 14:51 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x. 1 jenkins jenkins 2.6M May 30 14:51 libgmssl.so.3.0 -rwxr-xr-x. 1 jenkins jenkins 88M May 30 14:51 libtiflash_proxy.so -rwxr-xr-x. 1 jenkins jenkins 50M May 30 14:51 minio -rwxr-xr-x. 1 jenkins jenkins 33M May 30 14:51 pd-api-bench -rwxr-xr-x. 1 jenkins jenkins 43M May 30 14:51 pd-ctl -rwxr-xr-x. 1 jenkins jenkins 30M May 30 14:51 pd-heartbeat-bench -rwxr-xr-x. 1 jenkins jenkins 29M May 30 14:51 pd-recover -rwxr-xr-x. 1 jenkins jenkins 103M May 30 14:51 pd-server -rwxr-xr-x. 1 jenkins jenkins 25M May 30 14:51 pd-tso-bench -rwxr-xr-x. 1 jenkins jenkins 29M May 30 14:51 regions-dump drwxr-sr-x. 4 jenkins jenkins 4.0K May 30 14:51 share -rwxr-xr-x. 1 jenkins jenkins 29M May 30 14:51 stores-dump -rwxr-xr-x. 1 jenkins jenkins 133M May 30 14:51 sync_diff_inspector -rwxr-xr-x. 1 jenkins jenkins 199M May 30 14:51 tidb-server -rwxr-xr-x. 1 jenkins jenkins 377M May 30 14:51 tiflash -rwxr-xr-x. 1 jenkins jenkins 357M May 30 14:51 tikv-server [Pipeline] } Cache saved successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc) 3309751808 bytes in 101.63 secs (32566613 bytes/sec) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Tests) [Pipeline] parallel [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G00') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G01') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G02') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G03') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G04') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G05') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G06') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G07') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G08') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G09') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G10') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G11') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G12') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G13') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G14') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G15') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G16') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G17') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G00') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G01') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G02') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G03') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G04') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G05') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G06') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G07') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G08') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G09') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G10') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G11') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G12') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G13') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G14') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G15') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G16') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G17') [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f359n [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-29046 Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-qdr15 [Pipeline] { [Pipeline] node Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f6q5z Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-mwr1g [Pipeline] node Agent p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f359n is provisioned from template p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-hbk2q-c3npm --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "95fb6506bb482bc2e153537e3e89ad908aa7d0fb" jenkins/label: "p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-hbk2q" name: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f359n" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - command: - "bin/pulsar" - "standalone" image: "apachepulsar/pulsar:2.10.4" imagePullPolicy: "IfNotPresent" name: "pulsar" resources: limits: memory: "6Gi" cpu: "6" requests: memory: "4Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f359n" - name: "JENKINS_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f359n" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f359n in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test [Pipeline] { [Pipeline] checkout Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate The recommended git tool is: git [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-blflz Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-rbkd5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@4662da14; decorates RemoteLauncher[hudson.remoting.Channel@c85605a:JNLP4-connect connection from 10.233.70.123/10.233.70.123:40186] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Agent p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-qdr15 is provisioned from template p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-md6km-dzcd0 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "0f32acf63514c4e850033f8dd7a938fcba7bc3b8" jenkins/label: "p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-md6km" name: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-qdr15" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - command: - "bin/pulsar" - "standalone" image: "apachepulsar/pulsar:2.10.4" imagePullPolicy: "IfNotPresent" name: "pulsar" resources: limits: memory: "6Gi" cpu: "6" requests: memory: "4Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-qdr15" - name: "JENKINS_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-qdr15" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-qdr15 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test [Pipeline] { [Pipeline] checkout The recommended git tool is: git Agent p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f6q5z is provisioned from template p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-21vrp-zt0qk --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "16f06108d6ed38bd69783827ee78e5f5e635ae21" jenkins/label: "p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-21vrp" name: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f6q5z" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - command: - "bin/pulsar" - "standalone" image: "apachepulsar/pulsar:2.10.4" imagePullPolicy: "IfNotPresent" name: "pulsar" resources: limits: memory: "6Gi" cpu: "6" requests: memory: "4Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f6q5z" - name: "JENKINS_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f6q5z" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git Running on p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f6q5z in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5pr5v [Pipeline] podTemplate [Pipeline] { [Pipeline] { Agent p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-29046 is provisioned from template p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-5tsj1-th3fv --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "5460f96e5be1a2dcaa681a3ff7c5c3dcb212e42c" jenkins/label: "p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-5tsj1" name: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-29046" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - command: - "bin/pulsar" - "standalone" image: "apachepulsar/pulsar:2.10.4" imagePullPolicy: "IfNotPresent" name: "pulsar" resources: limits: memory: "6Gi" cpu: "6" requests: memory: "4Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-29046" - name: "JENKINS_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-29046" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Running on p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-29046 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test [Pipeline] node [Pipeline] { [Pipeline] checkout [Pipeline] checkout The recommended git tool is: git The recommended git tool is: git No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@3dcdd77a; decorates RemoteLauncher[hudson.remoting.Channel@249747e4:JNLP4-connect connection from 10.233.72.196/10.233.72.196:38736] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5sgr5 Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@784943a0; decorates RemoteLauncher[hudson.remoting.Channel@4134662e:JNLP4-connect connection from 10.233.70.222/10.233.70.222:46670] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" [Pipeline] node No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@2d82089a; decorates RemoteLauncher[hudson.remoting.Channel@2ad7529:JNLP4-connect connection from 10.233.69.167/10.233.69.167:49326] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] stage [Pipeline] { (Test) [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] cache Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Agent p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-blflz is provisioned from template p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-2kprl-dg7mz --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "180a61881a21535d8885a573b745284859a5439b" jenkins/label: "p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-2kprl" name: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-blflz" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - command: - "bin/pulsar" - "standalone" image: "apachepulsar/pulsar:2.10.4" imagePullPolicy: "IfNotPresent" name: "pulsar" resources: limits: memory: "6Gi" cpu: "6" requests: memory: "4Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-blflz" - name: "JENKINS_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-blflz" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-blflz in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sv1w0 Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" Agent p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-mwr1g is provisioned from template p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-pq8zk-42tx2 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "c7ea00d36b59a2a039f5cb8141a2acd260cdcb8c" jenkins/label: "p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-pq8zk" name: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-mwr1g" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - command: - "bin/pulsar" - "standalone" image: "apachepulsar/pulsar:2.10.4" imagePullPolicy: "IfNotPresent" name: "pulsar" resources: limits: memory: "6Gi" cpu: "6" requests: memory: "4Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-mwr1g" - name: "JENKINS_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-mwr1g" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-mwr1g in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 Agent p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5pr5v is provisioned from template p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-6bx2k-rcsq8 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "e7a670359a2039a6698ad6d956c6cdf26b540b60" jenkins/label: "p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-6bx2k" name: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5pr5v" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - command: - "bin/pulsar" - "standalone" image: "apachepulsar/pulsar:2.10.4" imagePullPolicy: "IfNotPresent" name: "pulsar" resources: limits: memory: "6Gi" cpu: "6" requests: memory: "4Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5pr5v" - name: "JENKINS_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5pr5v" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5pr5v in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5sgr5 is provisioned from template p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-m497v-khk9l --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "c7b8096bfdb80f796b77eb8e350fc36f3a1c6c81" jenkins/label: "p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-m497v" name: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5sgr5" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - command: - "bin/pulsar" - "standalone" image: "apachepulsar/pulsar:2.10.4" imagePullPolicy: "IfNotPresent" name: "pulsar" resources: limits: memory: "6Gi" cpu: "6" requests: memory: "4Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5sgr5" - name: "JENKINS_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5sgr5" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5sgr5 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test Agent p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg is provisioned from template p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-2dbls-x8qq0 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "ca5173af5ae55a6eea38e536b61bdeaa67df6f78" jenkins/label: "p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-2dbls" name: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - command: - "bin/pulsar" - "standalone" image: "apachepulsar/pulsar:2.10.4" imagePullPolicy: "IfNotPresent" name: "pulsar" resources: limits: memory: "6Gi" cpu: "6" requests: memory: "4Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg" - name: "JENKINS_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" Agent p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-rbkd5 is provisioned from template p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-wszjw-ll9qp --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "329d0ee851b10ff9c4775b9898f6d5a7ad170c4d" jenkins/label: "p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-wszjw" name: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-rbkd5" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - command: - "bin/pulsar" - "standalone" image: "apachepulsar/pulsar:2.10.4" imagePullPolicy: "IfNotPresent" name: "pulsar" resources: limits: memory: "6Gi" cpu: "6" requests: memory: "4Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-rbkd5" - name: "JENKINS_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-rbkd5" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-rbkd5 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sv1w0 is provisioned from template p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-qrzvp-p83jw --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "c3083bdc40d353e08f3d2abca7c0a4da4bae4a14" jenkins/label: "p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-qrzvp" name: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sv1w0" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - command: - "bin/pulsar" - "standalone" image: "apachepulsar/pulsar:2.10.4" imagePullPolicy: "IfNotPresent" name: "pulsar" resources: limits: memory: "6Gi" cpu: "6" requests: memory: "4Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sv1w0" - name: "JENKINS_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sv1w0" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sv1w0 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc) 3309751808 bytes in 22.18 secs (149228495 bytes/sec) [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] { [Pipeline] sh [Pipeline] node + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G01 Run cases: http_api http_api_tls api_v2 kafka_simple_basic kafka_simple_basic_avro kafka_simple_handle_key_only kafka_simple_handle_key_only_avro kafka_simple_claim_check kafka_simple_claim_check_avro canal_json_adapter_compatibility canal_json_basic multi_topics avro_basic canal_json_handle_key_only open_protocol_handle_key_only canal_json_claim_check open_protocol_claim_check canal_json_storage_basic canal_json_storage_partition_table multi_tables_ddl PROW_JOB_ID=f6375c57-288c-4d36-bd8b-6f8b308ffae9 JENKINS_NODE_COOKIE=50ae1bc2-d237-456e-861c-bbb2f1bf8e67 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/ GOLANG_VERSION=1.21.10 HOSTNAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f359n HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test","buildid":"1796069089735610368","prowjobid":"f6375c57-288c-4d36-bd8b-6f8b308ffae9","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#462 TEST_GROUP=G01 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089735610368 GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-hbk2q GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-3335c3054d39d880f4ab7664857409e24c98a0c490a94485ce4d2e2d0717ce44 NODE_LABELS=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f359n p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-hbk2q GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f359n GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=462 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/http_api/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:54:27 CST 2024] <<<<<< run test case http_api success! >>>>>> [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-179cx [Pipeline] node [Pipeline] node [Pipeline] { [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] { Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8f2lb Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-n5fjf [Pipeline] { Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-0gwt8 [Pipeline] checkout [Pipeline] { The recommended git tool is: git [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] { [Pipeline] checkout Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-s6w70 [Pipeline] checkout Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-k5r94 The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git The recommended git tool is: git [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@51ce2682; decorates RemoteLauncher[hudson.remoting.Channel@2c6da3cc:JNLP4-connect connection from 10.233.93.237/10.233.93.237:36608] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] { No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@6b62561a; decorates RemoteLauncher[hudson.remoting.Channel@4fe149:JNLP4-connect connection from 10.233.88.170/10.233.88.170:48244] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] cache Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@24c2d657; decorates RemoteLauncher[hudson.remoting.Channel@68eed738:JNLP4-connect connection from 10.233.106.31/10.233.106.31:54376] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@25b269bc; decorates RemoteLauncher[hudson.remoting.Channel@2bf244e9:JNLP4-connect connection from 10.233.107.200/10.233.107.200:59648] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@77ca46b4; decorates RemoteLauncher[hudson.remoting.Channel@2ddbe648:JNLP4-connect connection from 10.233.100.210/10.233.100.210:41070] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/http_api_tls/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:54:30 CST 2024] <<<<<< run test case http_api_tls success! >>>>>> Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@6407f282; decorates RemoteLauncher[hudson.remoting.Channel@56aaab75:JNLP4-connect connection from 10.233.126.17/10.233.126.17:40278] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@7e1f44d4; decorates RemoteLauncher[hudson.remoting.Channel@7efb62a3:JNLP4-connect connection from 10.233.67.67/10.233.67.67:57412] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-vldjs Agent p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-k5r94 is provisioned from template p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-z0mtv-7x63j --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "cb90cdea66c3288888520e120f0c2721fd8721b9" jenkins/label: "p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-z0mtv" name: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-k5r94" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - command: - "bin/pulsar" - "standalone" image: "apachepulsar/pulsar:2.10.4" imagePullPolicy: "IfNotPresent" name: "pulsar" resources: limits: memory: "6Gi" cpu: "6" requests: memory: "4Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-k5r94" - name: "JENKINS_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-k5r94" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Avoid second fetch > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Avoid second fetch > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/api_v2/run.sh using Sink-Type: pulsar... <<================= Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" Running on p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-k5r94 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test Agent p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-n5fjf is provisioned from template p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-jjg00-0s229 Avoid second fetch --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "50d2f8971dc8b6aa7f9bde48ecc6edba0432acd8" jenkins/label: "p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-jjg00" name: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-n5fjf" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - command: - "bin/pulsar" - "standalone" image: "apachepulsar/pulsar:2.10.4" imagePullPolicy: "IfNotPresent" name: "pulsar" resources: limits: memory: "6Gi" cpu: "6" requests: memory: "4Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-n5fjf" - name: "JENKINS_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-n5fjf" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Running on p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-n5fjf in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 Agent p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-0gwt8 is provisioned from template p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-cmwtb-1cc0k --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "375e68ceef821a31392997ed4d1044de0d43591b" jenkins/label: "p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-cmwtb" name: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-0gwt8" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - command: - "bin/pulsar" - "standalone" image: "apachepulsar/pulsar:2.10.4" imagePullPolicy: "IfNotPresent" name: "pulsar" resources: limits: memory: "6Gi" cpu: "6" requests: memory: "4Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-0gwt8" - name: "JENKINS_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-0gwt8" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 Running on p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-0gwt8 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test Agent p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-s6w70 is provisioned from template p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-mlzqb-1hrxm --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "6af05c3e077a6156a2c1ed1371d928c4c143d3d2" jenkins/label: "p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-mlzqb" name: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-s6w70" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - command: - "bin/pulsar" - "standalone" image: "apachepulsar/pulsar:2.10.4" imagePullPolicy: "IfNotPresent" name: "pulsar" resources: limits: memory: "6Gi" cpu: "6" requests: memory: "4Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-s6w70" - name: "JENKINS_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-s6w70" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" Running on p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-s6w70 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test Agent p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8f2lb is provisioned from template p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-29j3c-rcv2l Agent p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-179cx is provisioned from template p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-khh42-fv213 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "f145e0f66e2e072084781220fa48c6b17cf59809" --- apiVersion: "v1" kind: "Pod" metadata: annotations: jenkins/label: "p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-khh42" name: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-179cx" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" labels: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "644f3878a90e2911f5e84f7758580f84652e974f" jenkins/label: "p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-29j3c" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: name: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8f2lb" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - command: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: - "bin/pulsar" - "standalone" image: "apachepulsar/pulsar:2.10.4" imagePullPolicy: "IfNotPresent" name: "pulsar" memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: resources: limits: memory: "6Gi" cpu: "6" requests: memory: "4Gi" cpu: "4" - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - command: - "bin/pulsar" - "standalone" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false image: "apachepulsar/pulsar:2.10.4" imagePullPolicy: "IfNotPresent" name: "pulsar" resources: limits: memory: "6Gi" cpu: "6" - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true requests: memory: "4Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-179cx" - name: "JENKINS_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-179cx" - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8f2lb" - name: "JENKINS_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8f2lb" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" volumes: - emptyDir: medium: "" name: "workspace-volume" find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_simple_basic/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:54:36 CST 2024] <<<<<< run test case kafka_simple_basic success! >>>>>> Running on p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8f2lb in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test Running on p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-179cx in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test Agent p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-vldjs is provisioned from template p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-vdwm7-v869z --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "93479165b8d3da61e7d234dbe665b7a096c2cd35" jenkins/label: "p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-vdwm7" name: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-vldjs" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - command: - "bin/pulsar" - "standalone" image: "apachepulsar/pulsar:2.10.4" imagePullPolicy: "IfNotPresent" name: "pulsar" resources: limits: memory: "6Gi" cpu: "6" requests: memory: "4Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-vldjs" - name: "JENKINS_NAME" value: "p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-vldjs" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-vldjs in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_simple_basic_avro/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:54:39 CST 2024] <<<<<< run test case kafka_simple_basic_avro success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_simple_handle_key_only/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:54:42 CST 2024] <<<<<< run test case kafka_simple_handle_key_only success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_simple_handle_key_only_avro/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:54:45 CST 2024] <<<<<< run test case kafka_simple_handle_key_only_avro success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_simple_claim_check/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:54:49 CST 2024] <<<<<< run test case kafka_simple_claim_check success! >>>>>> Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc) 3309751808 bytes in 17.00 secs (194711390 bytes/sec) [Pipeline] { [Pipeline] cache find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_simple_claim_check_avro/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:54:52 CST 2024] <<<<<< run test case kafka_simple_claim_check_avro success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/canal_json_adapter_compatibility/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:54:55 CST 2024] <<<<<< run test case canal_json_adapter_compatibility success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/canal_json_basic/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/canal_json_basic Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc) 3309751808 bytes in 9.96 secs (332399824 bytes/sec) [Pipeline] { [Pipeline] cache Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212000900012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f359n, pid:1966, start at 2024-05-30 14:55:12.956796039 +0800 CST m=+5.466099521 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:57:12.964 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:55:12.932 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:45:12.932 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc) 3309751808 bytes in 7.46 secs (443486133 bytes/sec) [Pipeline] { [Pipeline] sh [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G02 Run cases: consistent_replicate_ddl consistent_replicate_gbk consistent_replicate_nfs consistent_replicate_storage_file consistent_replicate_storage_file_large_value consistent_replicate_storage_s3 consistent_partition_table kafka_big_txn_v2 kafka_big_messages_v2 multi_tables_ddl_v2 multi_topics_v2 storage_cleanup csv_storage_basic csv_storage_multi_tables_ddl csv_storage_partition_table PROW_JOB_ID=f6375c57-288c-4d36-bd8b-6f8b308ffae9 JENKINS_NODE_COOKIE=b35c0c24-8277-4ab5-8155-ca51260aa88c BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/ GOLANG_VERSION=1.21.10 HOSTNAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-qdr15 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test","buildid":"1796069089735610368","prowjobid":"f6375c57-288c-4d36-bd8b-6f8b308ffae9","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#462 TEST_GROUP=G02 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089735610368 GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-md6km GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-3335c3054d39d880f4ab7664857409e24c98a0c490a94485ce4d2e2d0717ce44 NODE_LABELS=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-md6km p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-qdr15 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-qdr15 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=462 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/consistent_replicate_ddl/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:55:15 CST 2024] <<<<<< run test case consistent_replicate_ddl success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] sh VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212000900012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f359n, pid:1966, start at 2024-05-30 14:55:12.956796039 +0800 CST m=+5.466099521 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:57:12.964 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:55:12.932 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:45:12.932 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212000340014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f359n, pid:2051, start at 2024-05-30 14:55:12.948987511 +0800 CST m=+5.368757419 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:57:12.956 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:55:12.958 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:45:12.958 +0800 All versions after safe point can be accessed. (DO NOT EDIT) + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G03 Run cases: row_format drop_many_tables processor_stop_delay partition_table PROW_JOB_ID=f6375c57-288c-4d36-bd8b-6f8b308ffae9 JENKINS_NODE_COOKIE=6b67992b-4506-48cf-bb62-3aed97db7c98 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/ GOLANG_VERSION=1.21.10 HOSTNAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f6q5z HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test","buildid":"1796069089735610368","prowjobid":"f6375c57-288c-4d36-bd8b-6f8b308ffae9","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#462 TEST_GROUP=G03 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089735610368 GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-21vrp GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-3335c3054d39d880f4ab7664857409e24c98a0c490a94485ce4d2e2d0717ce44 NODE_LABELS=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-21vrp p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f6q5z GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f6q5z GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=462 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/row_format/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/canal_json_basic/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/canal_json_basic/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/canal_json_basic/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/canal_json_basic/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/canal_json_basic/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G00 Run cases: bdr_mode capture_suicide_while_balance_table syncpoint hang_sink_suicide server_config_compatibility changefeed_dup_error_restart kafka_big_messages kafka_compression kafka_messages kafka_sink_error_resume mq_sink_lost_callback mq_sink_dispatcher kafka_column_selector kafka_column_selector_avro lossy_ddl storage_csv_update PROW_JOB_ID=f6375c57-288c-4d36-bd8b-6f8b308ffae9 JENKINS_NODE_COOKIE=fabd4c98-1c44-43d9-8763-06f0f2bfa3dc BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/ GOLANG_VERSION=1.21.10 HOSTNAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-29046 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test","buildid":"1796069089735610368","prowjobid":"f6375c57-288c-4d36-bd8b-6f8b308ffae9","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#462 TEST_GROUP=G00 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089735610368 GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-5tsj1 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-3335c3054d39d880f4ab7664857409e24c98a0c490a94485ce4d2e2d0717ce44 NODE_LABELS=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-29046 p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-5tsj1 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-29046 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=462 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/bdr_mode/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:55:16 CST 2024] <<<<<< run test case bdr_mode success! >>>>>> [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] checkout [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git The recommended git tool is: git [Pipeline] checkout [Pipeline] checkout [Pipeline] checkout The recommended git tool is: git The recommended git tool is: git [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Thu May 30 14:55:17 CST 2024] <<<<<< START cdc server in canal_json_basic case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_basic.33673369.out server --log-file /tmp/tidb_cdc_test/canal_json_basic/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/canal_json_basic/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/consistent_replicate_gbk/run.sh using Sink-Type: pulsar... <<================= * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:24927; Connection refused * Closing connection 0 [Pipeline] cache No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@3ccff502; decorates RemoteLauncher[hudson.remoting.Channel@6ea6bcca:JNLP4-connect connection from 10.233.88.74/10.233.88.74:43412] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@5e30d57d; decorates RemoteLauncher[hudson.remoting.Channel@d14a858:JNLP4-connect connection from 10.233.123.23/10.233.123.23:34752] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 You are running an older version of MinIO released 3 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend start tidb cluster in /tmp/tidb_cdc_test/row_format Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@3ecb516b; decorates RemoteLauncher[hudson.remoting.Channel@2f874d07:JNLP4-connect connection from 10.233.68.16/10.233.68.16:52400] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@4ee01832; decorates RemoteLauncher[hudson.remoting.Channel@7cbe14ec:JNLP4-connect connection from 10.233.72.29/10.233.72.29:43628] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@1478772b; decorates RemoteLauncher[hudson.remoting.Channel@1d97cc7:JNLP4-connect connection from 10.233.70.135/10.233.70.135:60222] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@7c97dcec; decorates RemoteLauncher[hudson.remoting.Channel@22b02fa6:JNLP4-connect connection from 10.233.69.50/10.233.69.50:53698] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/capture_suicide_while_balance_table/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:55:19 CST 2024] <<<<<< run test case capture_suicide_while_balance_table success! >>>>>> The recommended git tool is: git Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:24927 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 17D431CE63B0A191 < X-Xss-Protection: 1; mode=block < Date: Thu, 30 May 2024 06:55:20 GMT < { [data not shown] * Connection #0 to host 127.0.0.1 left intact Bucket 's3://logbucket/' created [Thu May 30 14:55:20 CST 2024] <<<<<< run test case consistent_replicate_gbk success! >>>>>> Exiting on signal: INTERRUPT No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@297dfc7b; decorates RemoteLauncher[hudson.remoting.Channel@256bd932:JNLP4-connect connection from 10.233.108.236/10.233.108.236:50578] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:55:20 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d159e84a-28be-4044-9102-ee31a5dbf984 {"id":"d159e84a-28be-4044-9102-ee31a5dbf984","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc847d2cfbc d159e84a-28be-4044-9102-ee31a5dbf984 /tidb/cdc/default/default/upstream/7374682631533901677 {"id":7374682631533901677,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d159e84a-28be-4044-9102-ee31a5dbf984 {"id":"d159e84a-28be-4044-9102-ee31a5dbf984","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc847d2cfbc d159e84a-28be-4044-9102-ee31a5dbf984 /tidb/cdc/default/default/upstream/7374682631533901677 {"id":7374682631533901677,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d159e84a-28be-4044-9102-ee31a5dbf984 {"id":"d159e84a-28be-4044-9102-ee31a5dbf984","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc847d2cfbc d159e84a-28be-4044-9102-ee31a5dbf984 /tidb/cdc/default/default/upstream/7374682631533901677 {"id":7374682631533901677,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_basic.cli.3410.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-canal-json-test?protocol=canal-json&enable-tidb-extension=true' > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" Create changefeed successfully! ID: 5921804c-df07-4584-affe-8f3698dd7a1b Info: {"upstream_id":7374682631533901677,"namespace":"default","id":"5921804c-df07-4584-affe-8f3698dd7a1b","sink_uri":"pulsar://127.0.0.1:6650/ticdc-canal-json-test?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T14:55:21.505112994+08:00","start_ts":450114911281348613,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114911281348613,"checkpoint_ts":450114911281348613,"checkpoint_time":"2024-05-30 14:55:21.282"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/syncpoint/run.sh using Sink-Type: pulsar... <<================= kafka downstream isn't support syncpoint record [Thu May 30 14:55:22 CST 2024] <<<<<< run test case syncpoint success! >>>>>> > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 + set +x Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" Avoid second fetch Checking out Revision e64357d6ddd9be700fd5c8d87859c826b78da53a (origin/main) > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 Commit message: "Define code ownership using "OWNERS" files in the central CI config repo (#2976)" =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/consistent_replicate_nfs/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:55:23 CST 2024] <<<<<< run test case consistent_replicate_nfs success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f e64357d6ddd9be700fd5c8d87859c826b78da53a # timeout=10 Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/hang_sink_suicide/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:55:25 CST 2024] <<<<<< run test case hang_sink_suicide success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/consistent_replicate_storage_file/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:55:26 CST 2024] <<<<<< run test case consistent_replicate_storage_file success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Thu May 30 14:55:27 CST 2024] <<<<<< START Pulsar consumer in canal_json_basic case >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/server_config_compatibility/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:55:28 CST 2024] <<<<<< run test case server_config_compatibility success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/consistent_replicate_storage_file_large_value/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:55:29 CST 2024] <<<<<< run test case consistent_replicate_storage_file_large_value success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2120fe500013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f6q5z, pid:1242, start at 2024-05-30 14:55:29.194011237 +0800 CST m=+5.228499924 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:57:29.205 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:55:29.172 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:45:29.172 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/changefeed_dup_error_restart/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:55:31 CST 2024] <<<<<< run test case changefeed_dup_error_restart success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2120fe500013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f6q5z, pid:1242, start at 2024-05-30 14:55:29.194011237 +0800 CST m=+5.228499924 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:57:29.205 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:55:29.172 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:45:29.172 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21210000000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f6q5z, pid:1324, start at 2024-05-30 14:55:29.2927837 +0800 CST m=+5.272352017 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:57:29.300 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:55:29.280 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:45:29.280 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/row_format/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/row_format/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/row_format/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/row_format/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/row_format/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table test.finish_mark not exists for 1-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/consistent_replicate_storage_s3/run.sh using Sink-Type: pulsar... <<================= * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:24927; Connection refused * Closing connection 0 You are running an older version of MinIO released 3 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.cli.2586.out cli tso query --pd=http://127.0.0.1:2379 table test.finish_mark not exists for 2-th check, retry later * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:24927 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 17D431D1AF9B0441 < X-Xss-Protection: 1; mode=block < Date: Thu, 30 May 2024 06:55:34 GMT < { [data not shown] * Connection #0 to host 127.0.0.1 left intact Bucket 's3://logbucket/' created [Thu May 30 14:55:34 CST 2024] <<<<<< run test case consistent_replicate_storage_s3 success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Exiting on signal: INTERRUPT find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_big_messages/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:55:34 CST 2024] <<<<<< run test case kafka_big_messages success! >>>>>> Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc) 3309751808 bytes in 14.50 secs (228260664 bytes/sec) [Pipeline] { [Pipeline] cache table test.finish_mark not exists for 3-th check, retry later + set +x + tso='450114914778611716 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114914778611716 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 14:55:36 CST 2024] <<<<<< START cdc server in row_format case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.26122614.out server --log-file /tmp/tidb_cdc_test/row_format/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/row_format/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/consistent_partition_table/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:55:37 CST 2024] <<<<<< run test case consistent_partition_table success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_compression/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:55:38 CST 2024] <<<<<< run test case kafka_compression success! >>>>>> table test.finish_mark not exists for 4-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:55:39 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7d6402da-ac16-4bd5-8a9e-22c7b8bf7568 {"id":"7d6402da-ac16-4bd5-8a9e-22c7b8bf7568","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8481754c2 7d6402da-ac16-4bd5-8a9e-22c7b8bf7568 /tidb/cdc/default/default/upstream/7374682701142584348 {"id":7374682701142584348,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7d6402da-ac16-4bd5-8a9e-22c7b8bf7568 {"id":"7d6402da-ac16-4bd5-8a9e-22c7b8bf7568","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8481754c2 7d6402da-ac16-4bd5-8a9e-22c7b8bf7568 /tidb/cdc/default/default/upstream/7374682701142584348 {"id":7374682701142584348,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7d6402da-ac16-4bd5-8a9e-22c7b8bf7568 {"id":"7d6402da-ac16-4bd5-8a9e-22c7b8bf7568","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8481754c2 7d6402da-ac16-4bd5-8a9e-22c7b8bf7568 /tidb/cdc/default/default/upstream/7374682701142584348 {"id":7374682701142584348,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.cli.2656.out cli changefeed create --start-ts=450114914778611716 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-row-format-test-29409?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: f187b51a-8926-449a-a8b4-185ca76536f4 Info: {"upstream_id":7374682701142584348,"namespace":"default","id":"f187b51a-8926-449a-a8b4-185ca76536f4","sink_uri":"pulsar://127.0.0.1:6650/ticdc-row-format-test-29409?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T14:55:39.870254652+08:00","start_ts":450114914778611716,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114914778611716,"checkpoint_ts":450114914778611716,"checkpoint_time":"2024-05-30 14:55:34.623"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_big_txn_v2/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:55:40 CST 2024] <<<<<< run test case kafka_big_txn_v2 success! >>>>>> table test.finish_mark not exists for 5-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_messages/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:55:41 CST 2024] <<<<<< run test case kafka_messages success! >>>>>> + set +x [Thu May 30 14:55:41 CST 2024] <<<<<< START Pulsar consumer in row_format case >>>>>> table test.finish_mark not exists for 6-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_big_messages_v2/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:55:43 CST 2024] <<<<<< run test case kafka_big_messages_v2 success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_sink_error_resume/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:55:44 CST 2024] <<<<<< run test case kafka_sink_error_resume success! >>>>>> table test.finish_mark not exists for 7-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc) 3309751808 bytes in 6.91 secs (479286580 bytes/sec) [Pipeline] { [Pipeline] cache table test.finish_mark not exists for 8-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/multi_tables_ddl_v2/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:55:46 CST 2024] <<<<<< run test case multi_tables_ddl_v2 success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/mq_sink_lost_callback/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:55:47 CST 2024] <<<<<< run test case mq_sink_lost_callback success! >>>>>> table test.finish_mark not exists for 9-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/multi_topics_v2/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:55:49 CST 2024] <<<<<< run test case multi_topics_v2 success! >>>>>> table test.finish_mark not exists for 10-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/mq_sink_dispatcher/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:55:50 CST 2024] <<<<<< run test case mq_sink_dispatcher success! >>>>>> table row_format.finish_mark not exists for 1-th check, retry later table test.finish_mark not exists for 11-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/storage_cleanup/run.sh using Sink-Type: pulsar... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/storage_cleanup/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/storage_cleanup ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/storage_cleanup + source /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/storage_cleanup/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/storage_cleanup + CDC_BINARY=cdc.test + SINK_TYPE=pulsar + EXIST_FILES=() + CLEANED_FILES=() + trap stop_tidb_cluster EXIT + run pulsar + '[' pulsar '!=' storage ']' + return + check_logs /tmp/tidb_cdc_test/storage_cleanup ++ date + echo '[Thu May 30 14:55:52 CST 2024] <<<<<< run test case storage_cleanup success! >>>>>>' [Thu May 30 14:55:52 CST 2024] <<<<<< run test case storage_cleanup success! >>>>>> + stop_tidb_cluster table row_format.finish_mark not exists for 2-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_column_selector/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:55:53 CST 2024] <<<<<< run test case kafka_column_selector success! >>>>>> Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc) 3309751808 bytes in 5.90 secs (561096606 bytes/sec) [Pipeline] { [Pipeline] cache table test.finish_mark not exists for 12-th check, retry later table row_format.finish_mark not exists for 3-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/csv_storage_basic/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:55:55 CST 2024] <<<<<< run test case csv_storage_basic success! >>>>>> table test.finish_mark not exists for 13-th check, retry later table row_format.finish_mark not exists for 4-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_column_selector_avro/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:55:56 CST 2024] <<<<<< run test case kafka_column_selector_avro success! >>>>>> table test.finish_mark not exists for 14-th check, retry later table row_format.finish_mark not exists for 5-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:55:59 CST 2024] <<<<<< run test case csv_storage_multi_tables_ddl success! >>>>>> table test.finish_mark not exists for 15-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/lossy_ddl/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:56:00 CST 2024] <<<<<< run test case lossy_ddl success! >>>>>> table row_format.finish_mark not exists for 6-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc) 3309751808 bytes in 5.35 secs (618652231 bytes/sec) [Pipeline] { [Pipeline] cache find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/csv_storage_partition_table/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:56:02 CST 2024] <<<<<< run test case csv_storage_partition_table success! >>>>>> table test.finish_mark not exists for 16-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/storage_csv_update/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:56:03 CST 2024] <<<<<< run test case storage_csv_update success! >>>>>> table row_format.finish_mark not exists for 7-th check, retry later table test.finish_mark not exists for 17-th check, retry later table row_format.finish_mark not exists for 8-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m \033[0;36m<<< Run all test success >>>\033[0m table test.finish_mark exists check diff successfully table row_format.finish_mark not exists for 9-th check, retry later table row_format.finish_mark not exists for 10-th check, retry later table test.finish_mark not exists for 1-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc) 3309751808 bytes in 6.98 secs (474252013 bytes/sec) [Pipeline] { [Pipeline] cache table row_format.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... table test.finish_mark not exists for 2-th check, retry later wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 14:56:12 CST 2024] <<<<<< run test case row_format success! >>>>>> table test.finish_mark not exists for 3-th check, retry later table test.finish_mark not exists for 4-th check, retry later table test.finish_mark not exists for 5-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc) 3309751808 bytes in 6.64 secs (498634442 bytes/sec) [Pipeline] { [Pipeline] cache table test.finish_mark not exists for 6-th check, retry later table test.finish_mark not exists for 7-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/drop_many_tables/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table test.finish_mark not exists for 8-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/drop_many_tables Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... table test.finish_mark not exists for 9-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table test.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 14:56:30 CST 2024] <<<<<< run test case canal_json_basic success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc) 3309751808 bytes in 10.26 secs (322674873 bytes/sec) [Pipeline] { [Pipeline] sh [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G05 Run cases: charset_gbk ddl_manager multi_source PROW_JOB_ID=f6375c57-288c-4d36-bd8b-6f8b308ffae9 JENKINS_NODE_COOKIE=9f03e947-1df7-4202-b31c-465bfe31e238 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/ GOLANG_VERSION=1.21.10 HOSTNAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-blflz HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test","buildid":"1796069089735610368","prowjobid":"f6375c57-288c-4d36-bd8b-6f8b308ffae9","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#462 TEST_GROUP=G05 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089735610368 GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-2kprl GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-3335c3054d39d880f4ab7664857409e24c98a0c490a94485ce4d2e2d0717ce44 NODE_LABELS=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-blflz p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-2kprl GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-blflz GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=462 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/charset_gbk/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G04 Run cases: foreign_key ddl_puller_lag ddl_only_block_related_table changefeed_auto_stop PROW_JOB_ID=f6375c57-288c-4d36-bd8b-6f8b308ffae9 JENKINS_NODE_COOKIE=bac304b1-7f68-41b8-ae74-112204eef733 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/ GOLANG_VERSION=1.21.10 HOSTNAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-mwr1g HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test","buildid":"1796069089735610368","prowjobid":"f6375c57-288c-4d36-bd8b-6f8b308ffae9","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#462 TEST_GROUP=G04 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089735610368 GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-pq8zk GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-3335c3054d39d880f4ab7664857409e24c98a0c490a94485ce4d2e2d0717ce44 NODE_LABELS=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-mwr1g p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-pq8zk GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-mwr1g GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=462 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/foreign_key/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G07 Run cases: kv_client_stream_reconnect cdc split_region PROW_JOB_ID=f6375c57-288c-4d36-bd8b-6f8b308ffae9 JENKINS_NODE_COOKIE=2fa8f254-cb0f-4b30-b2b7-4f45fa43c334 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/ GOLANG_VERSION=1.21.10 HOSTNAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5pr5v HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test","buildid":"1796069089735610368","prowjobid":"f6375c57-288c-4d36-bd8b-6f8b308ffae9","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#462 TEST_GROUP=G07 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089735610368 GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-6bx2k GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-3335c3054d39d880f4ab7664857409e24c98a0c490a94485ce4d2e2d0717ce44 NODE_LABELS=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-6bx2k p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5pr5v GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5pr5v GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=462 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kv_client_stream_reconnect/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G08 Run cases: processor_err_chan changefeed_reconstruct multi_capture PROW_JOB_ID=f6375c57-288c-4d36-bd8b-6f8b308ffae9 JENKINS_NODE_COOKIE=0f951b79-8071-4efb-abf3-f16b37f6e6ef BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/ GOLANG_VERSION=1.21.10 HOSTNAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5sgr5 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test","buildid":"1796069089735610368","prowjobid":"f6375c57-288c-4d36-bd8b-6f8b308ffae9","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#462 TEST_GROUP=G08 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089735610368 GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-m497v GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-3335c3054d39d880f4ab7664857409e24c98a0c490a94485ce4d2e2d0717ce44 NODE_LABELS=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5sgr5 p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-m497v GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5sgr5 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=462 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/processor_err_chan/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G09 Run cases: gc_safepoint changefeed_pause_resume cli savepoint synced_status PROW_JOB_ID=f6375c57-288c-4d36-bd8b-6f8b308ffae9 JENKINS_NODE_COOKIE=cd0343fe-fe39-4726-933d-3a0fa2440f48 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/ GOLANG_VERSION=1.21.10 HOSTNAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test","buildid":"1796069089735610368","prowjobid":"f6375c57-288c-4d36-bd8b-6f8b308ffae9","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#462 TEST_GROUP=G09 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089735610368 GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-2dbls GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-3335c3054d39d880f4ab7664857409e24c98a0c490a94485ce4d2e2d0717ce44 NODE_LABELS=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-2dbls GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=462 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/gc_safepoint/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G10 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212503a00012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f6q5z, pid:3761, start at 2024-05-30 14:56:35.069809552 +0800 CST m=+5.129727487 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:35.076 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:35.048 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:35.048 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212503a00012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f6q5z, pid:3761, start at 2024-05-30 14:56:35.069809552 +0800 CST m=+5.129727487 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:35.076 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:35.048 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:35.048 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212505140002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f6q5z, pid:3846, start at 2024-05-30 14:56:35.143150091 +0800 CST m=+5.151397661 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:35.150 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:35.141 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:35.141 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Run cases: default_value simple cdc_server_tips event_filter PROW_JOB_ID=f6375c57-288c-4d36-bd8b-6f8b308ffae9 JENKINS_NODE_COOKIE=31795375-d90b-4dc5-a934-5f030b9a2b40 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/ GOLANG_VERSION=1.21.10 HOSTNAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sv1w0 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test","buildid":"1796069089735610368","prowjobid":"f6375c57-288c-4d36-bd8b-6f8b308ffae9","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#462 TEST_GROUP=G10 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089735610368 GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-qrzvp GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-3335c3054d39d880f4ab7664857409e24c98a0c490a94485ce4d2e2d0717ce44 NODE_LABELS=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-qrzvp p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sv1w0 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sv1w0 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=462 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/default_value/run.sh using Sink-Type: pulsar... <<================= Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/drop_many_tables/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/drop_many_tables/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test start tidb cluster in /tmp/tidb_cdc_test/charset_gbk Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G06 Run cases: sink_retry changefeed_error ddl_sequence resourcecontrol PROW_JOB_ID=f6375c57-288c-4d36-bd8b-6f8b308ffae9 JENKINS_NODE_COOKIE=501d62d1-d19e-43f8-b480-54eb60e94410 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/ GOLANG_VERSION=1.21.10 HOSTNAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-rbkd5 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test","buildid":"1796069089735610368","prowjobid":"f6375c57-288c-4d36-bd8b-6f8b308ffae9","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#462 TEST_GROUP=G06 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089735610368 GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-wszjw GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-3335c3054d39d880f4ab7664857409e24c98a0c490a94485ce4d2e2d0717ce44 NODE_LABELS=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-wszjw p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-rbkd5 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-rbkd5 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=462 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/sink_retry/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/processor_err_chan Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc already exists) [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc already exists) start tidb cluster in /tmp/tidb_cdc_test/gc_safepoint Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.cli.5190.out cli tso query --pd=http://127.0.0.1:2379 [Pipeline] withEnv [Pipeline] { start tidb cluster in /tmp/tidb_cdc_test/kv_client_stream_reconnect Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/simple/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [Pipeline] // cache + set +x + tso='450114931536166913 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114931536166913 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 14:56:39 CST 2024] <<<<<< START cdc server in drop_many_tables case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.52215223.out server --log-file /tmp/tidb_cdc_test/drop_many_tables/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/drop_many_tables/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] // cache [Pipeline] } [Pipeline] } [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/multi_topics/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:56:39 CST 2024] <<<<<< run test case multi_topics success! >>>>>> [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [Pipeline] // dir [Pipeline] // dir [Pipeline] } [Pipeline] } [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { start tidb cluster in /tmp/tidb_cdc_test/foreign_key Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] cache start tidb cluster in /tmp/tidb_cdc_test/sink_retry Starting Upstream PD... Verifying downstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/avro_basic/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:56:42 CST 2024] <<<<<< run test case avro_basic success! >>>>>> Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:56:43 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/76edb7ca-45d6-4610-9fb4-4f0baea9da5a {"id":"76edb7ca-45d6-4610-9fb4-4f0baea9da5a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8491909be 76edb7ca-45d6-4610-9fb4-4f0baea9da5a /tidb/cdc/default/default/upstream/7374682986686827022 {"id":7374682986686827022,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/76edb7ca-45d6-4610-9fb4-4f0baea9da5a {"id":"76edb7ca-45d6-4610-9fb4-4f0baea9da5a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8491909be 76edb7ca-45d6-4610-9fb4-4f0baea9da5a /tidb/cdc/default/default/upstream/7374682986686827022 {"id":7374682986686827022,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/76edb7ca-45d6-4610-9fb4-4f0baea9da5a {"id":"76edb7ca-45d6-4610-9fb4-4f0baea9da5a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8491909be 76edb7ca-45d6-4610-9fb4-4f0baea9da5a /tidb/cdc/default/default/upstream/7374682986686827022 {"id":7374682986686827022,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.cli.5271.out cli changefeed create --start-ts=450114931536166913 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-drop-tables-test-28404?protocol=canal-json&enable-tidb-extension=true' Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Create changefeed successfully! ID: e8ffec67-03af-4333-af40-69709b3e9b67 Info: {"upstream_id":7374682986686827022,"namespace":"default","id":"e8ffec67-03af-4333-af40-69709b3e9b67","sink_uri":"pulsar://127.0.0.1:6650/ticdc-drop-tables-test-28404?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T14:56:43.526688003+08:00","start_ts":450114931536166913,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114931536166913,"checkpoint_ts":450114931536166913,"checkpoint_time":"2024-05-30 14:56:38.548"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... start tidb cluster in /tmp/tidb_cdc_test/simple Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x [Thu May 30 14:56:44 CST 2024] <<<<<< START Pulsar consumer in drop_many_tables case >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/canal_json_handle_key_only/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:56:45 CST 2024] <<<<<< run test case canal_json_handle_key_only success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table drop_tables.c not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table drop_tables.c not exists for 2-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2125c01c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:1177, start at 2024-05-30 14:56:47.143495866 +0800 CST m=+5.135012789 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:47.150 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:47.111 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:47.111 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2125c01c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:1177, start at 2024-05-30 14:56:47.143495866 +0800 CST m=+5.135012789 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:47.150 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:47.111 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:47.111 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2125c0800014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:1265, start at 2024-05-30 14:56:47.17547791 +0800 CST m=+5.086021027 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:47.182 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:47.185 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:47.185 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/gc_safepoint/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/gc_safepoint/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/open_protocol_handle_key_only/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:56:48 CST 2024] <<<<<< run test case open_protocol_handle_key_only success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2125cde80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5sgr5, pid:1255, start at 2024-05-30 14:56:48.039887537 +0800 CST m=+5.191361929 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:48.047 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:48.045 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:48.045 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2125cde80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5sgr5, pid:1255, start at 2024-05-30 14:56:48.039887537 +0800 CST m=+5.191361929 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:48.047 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:48.045 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:48.045 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2125cf780015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5sgr5, pid:1338, start at 2024-05-30 14:56:48.13279456 +0800 CST m=+5.196518885 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:48.139 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:48.144 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:48.144 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_err_chan/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_err_chan/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table drop_tables.c not exists for 3-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc) 3309751808 bytes in 7.57 secs (437049526 bytes/sec) [Pipeline] { [Pipeline] cache [Thu May 30 14:56:50 CST 2024] <<<<<< START cdc server in gc_safepoint case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/txnutil/gc/InjectGcSafepointUpdateInterval=return(500)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.gc_safepoint.25682570.out server --log-file /tmp/tidb_cdc_test/gc_safepoint/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/gc_safepoint/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2125d0080012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-blflz, pid:1202, start at 2024-05-30 14:56:48.152573891 +0800 CST m=+5.270845875 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:48.158 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:48.130 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:48.130 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2125d0080012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-blflz, pid:1202, start at 2024-05-30 14:56:48.152573891 +0800 CST m=+5.270845875 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:48.158 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:48.130 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:48.130 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2125d1800015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-blflz, pid:1272, start at 2024-05-30 14:56:48.271212286 +0800 CST m=+5.324646991 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:48.278 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:48.274 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:48.274 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/charset_gbk/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/charset_gbk/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2125f7000016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-mwr1g, pid:1200, start at 2024-05-30 14:56:50.672938375 +0800 CST m=+5.161279490 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:50.680 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:50.674 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:50.674 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2125f7000016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-mwr1g, pid:1200, start at 2024-05-30 14:56:50.672938375 +0800 CST m=+5.161279490 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:50.680 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:50.674 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:50.674 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2125f9440003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-mwr1g, pid:1293, start at 2024-05-30 14:56:50.773078941 +0800 CST m=+5.206745067 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:50.781 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:50.769 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:50.769 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/foreign_key/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/foreign_key/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/canal_json_claim_check/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:56:51 CST 2024] <<<<<< run test case canal_json_claim_check success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table drop_tables.c not exists for 4-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:56:53 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/37470fbc-4726-4b05-a30d-28fa8ac06917 {"id":"37470fbc-4726-4b05-a30d-28fa8ac06917","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8494e07b8 37470fbc-4726-4b05-a30d-28fa8ac06917 /tidb/cdc/default/default/upstream/7374683036886642251 {"id":7374683036886642251,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/37470fbc-4726-4b05-a30d-28fa8ac06917 {"id":"37470fbc-4726-4b05-a30d-28fa8ac06917","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8494e07b8 37470fbc-4726-4b05-a30d-28fa8ac06917 /tidb/cdc/default/default/upstream/7374683036886642251 {"id":7374683036886642251,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/37470fbc-4726-4b05-a30d-28fa8ac06917 {"id":"37470fbc-4726-4b05-a30d-28fa8ac06917","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8494e07b8 37470fbc-4726-4b05-a30d-28fa8ac06917 /tidb/cdc/default/default/upstream/7374683036886642251 {"id":7374683036886642251,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Thu May 30 14:56:53 CST 2024] <<<<<< START cdc server in charset_gbk case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.charset_gbk.26162618.out server --log-file /tmp/tidb_cdc_test/charset_gbk/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/charset_gbk/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Thu May 30 14:56:53 CST 2024] <<<<<< START Pulsar consumer in gc_safepoint case >>>>>> 0 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 1-th time, retry later table drop_tables.c not exists for 5-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.cli.2602.out cli tso query --pd=http://127.0.0.1:2379 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/open_protocol_claim_check/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:56:54 CST 2024] <<<<<< run test case open_protocol_claim_check success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212605c40013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5pr5v, pid:1138, start at 2024-05-30 14:56:51.604530294 +0800 CST m=+7.540478953 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:51.612 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:51.619 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:51.619 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212605c40013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5pr5v, pid:1138, start at 2024-05-30 14:56:51.604530294 +0800 CST m=+7.540478953 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:51.612 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:51.619 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:51.619 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212607480013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5pr5v, pid:1213, start at 2024-05-30 14:56:51.688420041 +0800 CST m=+7.548011071 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:51.695 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:51.666 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:51.666 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + tso='450114935632429057 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114935632429057 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 14:56:55 CST 2024] <<<<<< START cdc server in foreign_key case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.26372639.out server --log-file /tmp/tidb_cdc_test/foreign_key/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/foreign_key/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table drop_tables.c exists check diff successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:56:56 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/01bd3664-38ad-498f-9cf6-6ab4ee7afcc2 {"id":"01bd3664-38ad-498f-9cf6-6ab4ee7afcc2","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849499cbb 01bd3664-38ad-498f-9cf6-6ab4ee7afcc2 /tidb/cdc/default/default/upstream/7374683043026975196 {"id":7374683043026975196,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/01bd3664-38ad-498f-9cf6-6ab4ee7afcc2 {"id":"01bd3664-38ad-498f-9cf6-6ab4ee7afcc2","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849499cbb 01bd3664-38ad-498f-9cf6-6ab4ee7afcc2 /tidb/cdc/default/default/upstream/7374683043026975196 {"id":7374683043026975196,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/01bd3664-38ad-498f-9cf6-6ab4ee7afcc2 {"id":"01bd3664-38ad-498f-9cf6-6ab4ee7afcc2","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849499cbb 01bd3664-38ad-498f-9cf6-6ab4ee7afcc2 /tidb/cdc/default/default/upstream/7374683043026975196 {"id":7374683043026975196,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: ea9a1fc5-8b80-4875-8320-304f69abf339 Info: {"upstream_id":7374683043026975196,"namespace":"default","id":"ea9a1fc5-8b80-4875-8320-304f69abf339","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-30T14:56:56.62007557+08:00","start_ts":450114935398334471,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114935398334471,"checkpoint_ts":450114935398334471,"checkpoint_time":"2024-05-30 14:56:53.281"} [Thu May 30 14:56:56 CST 2024] <<<<<< START Pulsar consumer in charset_gbk case >>>>>> check diff failed 2-th time, retry later wait process cdc.test exit for 1-th time... [Thu May 30 14:56:56 CST 2024] <<<<<< START cdc server in kv_client_stream_reconnect case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/kv/kvClientForceReconnect=return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kv_client_stream_reconnect.25952597.out server --log-file /tmp/tidb_cdc_test/kv_client_stream_reconnect/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kv_client_stream_reconnect/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 14:56:57 CST 2024] <<<<<< run test case drop_many_tables success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21263b340012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sv1w0, pid:1295, start at 2024-05-30 14:56:55.029165316 +0800 CST m=+5.794295129 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:55.041 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:55.038 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:55.038 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21263b340012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sv1w0, pid:1295, start at 2024-05-30 14:56:55.029165316 +0800 CST m=+5.794295129 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:55.041 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:55.038 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:55.038 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21263c70000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sv1w0, pid:1360, start at 2024-05-30 14:56:55.087655591 +0800 CST m=+5.791857218 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:55.098 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:55.068 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:55.068 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/canal_json_storage_basic/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:56:57 CST 2024] <<<<<< run test case canal_json_storage_basic success! >>>>>> Logging trace to /tmp/tidb_cdc_test/simple/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/simple/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/simple/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/simple/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/simple/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff successfully check_safepoint_forward http://127.0.0.1:2379 7374683036886642251 450114936388976648 450114935458365449 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:56:58 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d0a0096e-47db-42f4-bb10-9d9b3b049241 {"id":"d0a0096e-47db-42f4-bb10-9d9b3b049241","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84957d8be d0a0096e-47db-42f4-bb10-9d9b3b049241 /tidb/cdc/default/default/upstream/7374683054840977710 {"id":7374683054840977710,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d0a0096e-47db-42f4-bb10-9d9b3b049241 {"id":"d0a0096e-47db-42f4-bb10-9d9b3b049241","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84957d8be d0a0096e-47db-42f4-bb10-9d9b3b049241 /tidb/cdc/default/default/upstream/7374683054840977710 {"id":7374683054840977710,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d0a0096e-47db-42f4-bb10-9d9b3b049241 {"id":"d0a0096e-47db-42f4-bb10-9d9b3b049241","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84957d8be d0a0096e-47db-42f4-bb10-9d9b3b049241 /tidb/cdc/default/default/upstream/7374683054840977710 {"id":7374683054840977710,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.cli.2696.out cli changefeed create --start-ts=450114935632429057 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-foreign-key-test-4230?protocol=canal-json&enable-tidb-extension=true' [Thu May 30 14:56:58 CST 2024] <<<<<< START cdc server in processor_err_chan case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/sourcemanager/puller/ProcessorAddTableError=1*return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_err_chan.27122714.out server --log-file /tmp/tidb_cdc_test/processor_err_chan/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_err_chan/cdc_data --cluster-id default --config /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/processor_err_chan/conf/server.toml --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Create changefeed successfully! ID: 5dc4a637-d623-42b6-bb74-9d58e4e7965d Info: {"upstream_id":7374683054840977710,"namespace":"default","id":"5dc4a637-d623-42b6-bb74-9d58e4e7965d","sink_uri":"pulsar://127.0.0.1:6650/ticdc-foreign-key-test-4230?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T14:56:59.182985271+08:00","start_ts":450114935632429057,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114935632429057,"checkpoint_ts":450114935632429057,"checkpoint_time":"2024-05-30 14:56:54.174"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... run task successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:56:59 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/829a566c-a925-46f2-95fb-616a493e23b4 {"id":"829a566c-a925-46f2-95fb-616a493e23b4","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8494fffc9 829a566c-a925-46f2-95fb-616a493e23b4 /tidb/cdc/default/default/upstream/7374683044299719173 {"id":7374683044299719173,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/829a566c-a925-46f2-95fb-616a493e23b4 {"id":"829a566c-a925-46f2-95fb-616a493e23b4","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8494fffc9 829a566c-a925-46f2-95fb-616a493e23b4 /tidb/cdc/default/default/upstream/7374683044299719173 {"id":7374683044299719173,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/829a566c-a925-46f2-95fb-616a493e23b4 {"id":"829a566c-a925-46f2-95fb-616a493e23b4","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8494fffc9 829a566c-a925-46f2-95fb-616a493e23b4 /tidb/cdc/default/default/upstream/7374683044299719173 {"id":7374683044299719173,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.cli.2610.out cli tso query --pd=http://127.0.0.1:2379 check_changefeed_state http://127.0.0.1:2379 f7bfd530-8777-47e2-86ec-9e5187eeb952 stopped null + endpoints=http://127.0.0.1:2379 + changefeed_id=f7bfd530-8777-47e2-86ec-9e5187eeb952 + expected_state=stopped + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c f7bfd530-8777-47e2-86ec-9e5187eeb952 -s + info='{ "upstream_id": 7374683036886642251, "namespace": "default", "id": "f7bfd530-8777-47e2-86ec-9e5187eeb952", "state": "stopped", "checkpoint_tso": 450114936913264648, "checkpoint_time": "2024-05-30 14:56:59.060", "error": null }' + echo '{ "upstream_id": 7374683036886642251, "namespace": "default", "id": "f7bfd530-8777-47e2-86ec-9e5187eeb952", "state": "stopped", "checkpoint_tso": 450114936913264648, "checkpoint_time": "2024-05-30 14:56:59.060", "error": null }' { "upstream_id": 7374683036886642251, "namespace": "default", "id": "f7bfd530-8777-47e2-86ec-9e5187eeb952", "state": "stopped", "checkpoint_tso": 450114936913264648, "checkpoint_time": "2024-05-30 14:56:59.060", "error": null } ++ echo '{' '"upstream_id":' 7374683036886642251, '"namespace":' '"default",' '"id":' '"f7bfd530-8777-47e2-86ec-9e5187eeb952",' '"state":' '"stopped",' '"checkpoint_tso":' 450114936913264648, '"checkpoint_time":' '"2024-05-30' '14:56:59.060",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7374683036886642251, '"namespace":' '"default",' '"id":' '"f7bfd530-8777-47e2-86ec-9e5187eeb952",' '"state":' '"stopped",' '"checkpoint_tso":' 450114936913264648, '"checkpoint_time":' '"2024-05-30' '14:56:59.060",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_equal http://127.0.0.1:2379 7374683036886642251 [Thu May 30 14:56:59 CST 2024] <<<<<< START Pulsar consumer in kv_client_stream_reconnect case >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/canal_json_storage_partition_table/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:57:00 CST 2024] <<<<<< run test case canal_json_storage_partition_table success! >>>>>> + set +x [Thu May 30 14:57:00 CST 2024] <<<<<< START Pulsar consumer in foreign_key case >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212658240014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-rbkd5, pid:1231, start at 2024-05-30 14:56:56.888139297 +0800 CST m=+7.536855914 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:56.896 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:56.890 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:56.890 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212658240014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-rbkd5, pid:1231, start at 2024-05-30 14:56:56.888139297 +0800 CST m=+7.536855914 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:56.896 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:56.890 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:56.890 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212658580013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-rbkd5, pid:1314, start at 2024-05-30 14:56:56.889459831 +0800 CST m=+7.455588375 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:58:56.900 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:56:56.904 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:46:56.904 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sink_retry/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sink_retry/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + set +x + tso='450114937143689217 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114937143689217 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:01 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fc5366e1-27b4-49fc-a18f-b55c6f74dd9c {"id":"fc5366e1-27b4-49fc-a18f-b55c6f74dd9c","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8494d5e65 fc5366e1-27b4-49fc-a18f-b55c6f74dd9c /tidb/cdc/default/default/upstream/7374683041280776857 {"id":7374683041280776857,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fc5366e1-27b4-49fc-a18f-b55c6f74dd9c {"id":"fc5366e1-27b4-49fc-a18f-b55c6f74dd9c","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8494d5e65 fc5366e1-27b4-49fc-a18f-b55c6f74dd9c /tidb/cdc/default/default/upstream/7374683041280776857 {"id":7374683041280776857,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fc5366e1-27b4-49fc-a18f-b55c6f74dd9c {"id":"fc5366e1-27b4-49fc-a18f-b55c6f74dd9c","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8494d5e65 fc5366e1-27b4-49fc-a18f-b55c6f74dd9c /tidb/cdc/default/default/upstream/7374683041280776857 {"id":7374683041280776857,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Thu May 30 14:57:02 CST 2024] <<<<<< START Pulsar consumer in processor_err_chan case >>>>>> check_changefeed_state http://127.0.0.1:2379 b9169167-42da-4015-ab7a-f4034330e984 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=b9169167-42da-4015-ab7a-f4034330e984 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c b9169167-42da-4015-ab7a-f4034330e984 -s + info='{ "upstream_id": 7374683041280776857, "namespace": "default", "id": "b9169167-42da-4015-ab7a-f4034330e984", "state": "normal", "checkpoint_tso": 450114937629966339, "checkpoint_time": "2024-05-30 14:57:01.794", "error": null }' + echo '{ "upstream_id": 7374683041280776857, "namespace": "default", "id": "b9169167-42da-4015-ab7a-f4034330e984", "state": "normal", "checkpoint_tso": 450114937629966339, "checkpoint_time": "2024-05-30 14:57:01.794", "error": null }' { "upstream_id": 7374683041280776857, "namespace": "default", "id": "b9169167-42da-4015-ab7a-f4034330e984", "state": "normal", "checkpoint_tso": 450114937629966339, "checkpoint_time": "2024-05-30 14:57:01.794", "error": null } ++ echo '{' '"upstream_id":' 7374683041280776857, '"namespace":' '"default",' '"id":' '"b9169167-42da-4015-ab7a-f4034330e984",' '"state":' '"normal",' '"checkpoint_tso":' 450114937629966339, '"checkpoint_time":' '"2024-05-30' '14:57:01.794",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374683041280776857, '"namespace":' '"default",' '"id":' '"b9169167-42da-4015-ab7a-f4034330e984",' '"state":' '"normal",' '"checkpoint_tso":' 450114937629966339, '"checkpoint_time":' '"2024-05-30' '14:57:01.794",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully [Thu May 30 14:57:02 CST 2024] <<<<<< START cdc server in simple case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.26622664.out server --log-file /tmp/tidb_cdc_test/simple/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/simple/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 1-th time, retry later run task successfully table foreign_key.finish_mark not exists for 1-th check, retry later check_changefeed_state http://127.0.0.1:2379 f7bfd530-8777-47e2-86ec-9e5187eeb952 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=f7bfd530-8777-47e2-86ec-9e5187eeb952 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c f7bfd530-8777-47e2-86ec-9e5187eeb952 -s + info='{ "upstream_id": 7374683036886642251, "namespace": "default", "id": "f7bfd530-8777-47e2-86ec-9e5187eeb952", "state": "normal", "checkpoint_tso": 450114936913264648, "checkpoint_time": "2024-05-30 14:56:59.060", "error": null }' + echo '{ "upstream_id": 7374683036886642251, "namespace": "default", "id": "f7bfd530-8777-47e2-86ec-9e5187eeb952", "state": "normal", "checkpoint_tso": 450114936913264648, "checkpoint_time": "2024-05-30 14:56:59.060", "error": null }' { "upstream_id": 7374683036886642251, "namespace": "default", "id": "f7bfd530-8777-47e2-86ec-9e5187eeb952", "state": "normal", "checkpoint_tso": 450114936913264648, "checkpoint_time": "2024-05-30 14:56:59.060", "error": null } ++ echo '{' '"upstream_id":' 7374683036886642251, '"namespace":' '"default",' '"id":' '"f7bfd530-8777-47e2-86ec-9e5187eeb952",' '"state":' '"normal",' '"checkpoint_tso":' 450114936913264648, '"checkpoint_time":' '"2024-05-30' '14:56:59.060",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374683036886642251, '"namespace":' '"default",' '"id":' '"f7bfd530-8777-47e2-86ec-9e5187eeb952",' '"state":' '"normal",' '"checkpoint_tso":' 450114936913264648, '"checkpoint_time":' '"2024-05-30' '14:56:59.060",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_forward http://127.0.0.1:2379 7374683036886642251 450114936913264647 450114936913264648 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/multi_tables_ddl/run.sh using Sink-Type: pulsar... <<================= * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:24927; Connection refused * Closing connection 0 You are running an older version of MinIO released 3 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.cli.2561.out cli tso query --pd=http://127.0.0.1:2379 Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide run task successfully check_changefeed_state http://127.0.0.1:2379 f7bfd530-8777-47e2-86ec-9e5187eeb952 stopped null + endpoints=http://127.0.0.1:2379 + changefeed_id=f7bfd530-8777-47e2-86ec-9e5187eeb952 + expected_state=stopped + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c f7bfd530-8777-47e2-86ec-9e5187eeb952 -s + set +x + tso='450114938022395905 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + info='{ "upstream_id": 7374683036886642251, "namespace": "default", + echo 450114938022395905 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x "id": "f7bfd530-8777-47e2-86ec-9e5187eeb952", "state": "stopped", "checkpoint_tso": 450114936913264648, "checkpoint_time": "2024-05-30 14:56:59.060", "error": null }' + echo '{ "upstream_id": 7374683036886642251, "namespace": "default", "id": "f7bfd530-8777-47e2-86ec-9e5187eeb952", "state": "stopped", "checkpoint_tso": 450114936913264648, "checkpoint_time": "2024-05-30 14:56:59.060", "error": null }' { "upstream_id": 7374683036886642251, "namespace": "default", "id": "f7bfd530-8777-47e2-86ec-9e5187eeb952", "state": "stopped", "checkpoint_tso": 450114936913264648, "checkpoint_time": "2024-05-30 14:56:59.060", "error": null } ++ echo '{' '"upstream_id":' 7374683036886642251, '"namespace":' '"default",' '"id":' '"f7bfd530-8777-47e2-86ec-9e5187eeb952",' '"state":' '"stopped",' '"checkpoint_tso":' 450114936913264648, '"checkpoint_time":' '"2024-05-30' '14:56:59.060",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7374683036886642251, '"namespace":' '"default",' '"id":' '"f7bfd530-8777-47e2-86ec-9e5187eeb952",' '"state":' '"stopped",' '"checkpoint_tso":' 450114936913264648, '"checkpoint_time":' '"2024-05-30' '14:56:59.060",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully table foreign_key.finish_mark not exists for 2-th check, retry later * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:24927 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 17D431E6B97B2FDE < X-Xss-Protection: 1; mode=block < Date: Thu, 30 May 2024 06:57:04 GMT < { [data not shown] * Connection #0 to host 127.0.0.1 left intact check_changefeed_state http://127.0.0.1:2379 6761d21b-882c-4201-9eaa-f557710ac51b normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=6761d21b-882c-4201-9eaa-f557710ac51b + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 6761d21b-882c-4201-9eaa-f557710ac51b -s + info='{ "upstream_id": 7374683036886642251, "namespace": "default", "id": "6761d21b-882c-4201-9eaa-f557710ac51b", "state": "normal", "checkpoint_tso": 450114938446807044, "checkpoint_time": "2024-05-30 14:57:04.910", "error": null }' + echo '{ "upstream_id": 7374683036886642251, "namespace": "default", "id": "6761d21b-882c-4201-9eaa-f557710ac51b", "state": "normal", "checkpoint_tso": 450114938446807044, "checkpoint_time": "2024-05-30 14:57:04.910", "error": null }' { "upstream_id": 7374683036886642251, "namespace": "default", "id": "6761d21b-882c-4201-9eaa-f557710ac51b", "state": "normal", "checkpoint_tso": 450114938446807044, "checkpoint_time": "2024-05-30 14:57:04.910", "error": null } ++ echo '{' '"upstream_id":' 7374683036886642251, '"namespace":' '"default",' '"id":' '"6761d21b-882c-4201-9eaa-f557710ac51b",' '"state":' '"normal",' '"checkpoint_tso":' 450114938446807044, '"checkpoint_time":' '"2024-05-30' '14:57:04.910",' '"error":' null '}' ++ jq -r .state check diff failed 2-th time, retry later Bucket 's3://logbucket/' created Exiting on signal: INTERRUPT + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374683036886642251, '"namespace":' '"default",' '"id":' '"6761d21b-882c-4201-9eaa-f557710ac51b",' '"state":' '"normal",' '"checkpoint_tso":' 450114938446807044, '"checkpoint_time":' '"2024-05-30' '14:57:04.910",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_equal http://127.0.0.1:2379 7374683036886642251 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:05 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/29cc3cc9-d6de-4109-9b1f-c77dceb07a82 {"id":"29cc3cc9-d6de-4109-9b1f-c77dceb07a82","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849607be1 29cc3cc9-d6de-4109-9b1f-c77dceb07a82 /tidb/cdc/default/default/upstream/7374683073770588166 {"id":7374683073770588166,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/29cc3cc9-d6de-4109-9b1f-c77dceb07a82 {"id":"29cc3cc9-d6de-4109-9b1f-c77dceb07a82","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849607be1 29cc3cc9-d6de-4109-9b1f-c77dceb07a82 /tidb/cdc/default/default/upstream/7374683073770588166 {"id":7374683073770588166,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/29cc3cc9-d6de-4109-9b1f-c77dceb07a82 {"id":"29cc3cc9-d6de-4109-9b1f-c77dceb07a82","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849607be1 29cc3cc9-d6de-4109-9b1f-c77dceb07a82 /tidb/cdc/default/default/upstream/7374683073770588166 {"id":7374683073770588166,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.cli.2704.out cli changefeed create --start-ts=450114937143689217 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-simple-test-16289?protocol=canal-json&enable-tidb-extension=true' ***************** properties ***************** "recordcount"="10" "readallfields"="true" "workload"="core" "mysql.user"="root" "readproportion"="0" "requestdistribution"="uniform" "mysql.host"="127.0.0.1" "mysql.port"="4000" "dotransactions"="false" "updateproportion"="0" "operationcount"="0" "mysql.db"="sink_retry" "insertproportion"="0" "scanproportion"="0" "threadcount"="2" ********************************************** Run finished, takes 9.020088ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2334.8, Avg(us): 1739, Min(us): 847, Max(us): 4623, 95th(us): 5000, 99th(us): 5000 [Thu May 30 14:57:06 CST 2024] <<<<<< START cdc server in sink_retry case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkTxnRandomError=25%return(true)' + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.26162618.out server --log-file /tmp/tidb_cdc_test/sink_retry/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sink_retry/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Create changefeed successfully! ID: 50fdbe37-c5b4-454a-b3f8-e5fdfddf812c Info: {"upstream_id":7374683073770588166,"namespace":"default","id":"50fdbe37-c5b4-454a-b3f8-e5fdfddf812c","sink_uri":"pulsar://127.0.0.1:6650/ticdc-simple-test-16289?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T14:57:06.068199589+08:00","start_ts":450114937143689217,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114937143689217,"checkpoint_ts":450114937143689217,"checkpoint_time":"2024-05-30 14:56:59.939"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... check diff successfully =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/processor_stop_delay/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... wait process cdc.test exit for 1-th time... + set +x [Thu May 30 14:57:07 CST 2024] <<<<<< START Pulsar consumer in simple case >>>>>> succeed to verify meta placement rules ERROR 1146 (42S02) at line 1: Table 'test.simple1' doesn't exist check data failed 1-th time, retry later table foreign_key.finish_mark not exists for 3-th check, retry later wait process cdc.test exit for 2-th time... \033[0;36m<<< Run all test success >>>\033[0m wait process cdc.test exit for 3-th time... run task successfully Changefeed remove successfully. ID: f7bfd530-8777-47e2-86ec-9e5187eeb952 CheckpointTs: 450114936913264648 SinkURI: pulsar://127.0.0.1:6650/ticdc-gc-safepoint-13108?protocol=canal-json&enable-tidb-extension=true check_safepoint_forward http://127.0.0.1:2379 7374683036886642251 450114938446807043 450114938446807044 450114936913264648 cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 14:57:09 CST 2024] <<<<<< run test case processor_err_chan success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:09 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/22a13332-849c-4910-bcf2-114a03f9c3b7 {"id":"22a13332-849c-4910-bcf2-114a03f9c3b7","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8495a3eef 22a13332-849c-4910-bcf2-114a03f9c3b7 /tidb/cdc/default/default/upstream/7374683063960869405 {"id":7374683063960869405,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/22a13332-849c-4910-bcf2-114a03f9c3b7 {"id":"22a13332-849c-4910-bcf2-114a03f9c3b7","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8495a3eef 22a13332-849c-4910-bcf2-114a03f9c3b7 /tidb/cdc/default/default/upstream/7374683063960869405 {"id":7374683063960869405,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/22a13332-849c-4910-bcf2-114a03f9c3b7 {"id":"22a13332-849c-4910-bcf2-114a03f9c3b7","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8495a3eef 22a13332-849c-4910-bcf2-114a03f9c3b7 /tidb/cdc/default/default/upstream/7374683063960869405 {"id":7374683063960869405,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.cli.2667.out cli changefeed create --start-ts=450114938022395905 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-sink-retry-test-32000?protocol=canal-json&enable-tidb-extension=true' TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 2-th time, retry later run task successfully start tidb cluster in /tmp/tidb_cdc_test/processor_stop_delay Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... table foreign_key.finish_mark not exists for 4-th check, retry later Create changefeed successfully! ID: 86c8106d-11a8-4e2d-b1c8-e3e2e5688f29 Info: {"upstream_id":7374683063960869405,"namespace":"default","id":"86c8106d-11a8-4e2d-b1c8-e3e2e5688f29","sink_uri":"pulsar://127.0.0.1:6650/ticdc-sink-retry-test-32000?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T14:57:09.908398781+08:00","start_ts":450114938022395905,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114938022395905,"checkpoint_ts":450114938022395905,"checkpoint_time":"2024-05-30 14:57:03.291"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... Changefeed remove successfully. ID: 6761d21b-882c-4201-9eaa-f557710ac51b CheckpointTs: 450114939547811841 SinkURI: pulsar://127.0.0.1:6650/ticdc-gc-safepoint-13108?protocol=canal-json&enable-tidb-extension=true check_safepoint_cleared http://127.0.0.1:2379 7374683036886642251 run task successfully wait process cdc.test exit for 1-th time... + set +x [Thu May 30 14:57:11 CST 2024] <<<<<< START Pulsar consumer in sink_retry case >>>>>> wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 14:57:11 CST 2024] <<<<<< run test case gc_safepoint success! >>>>>> TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 3-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table foreign_key.finish_mark not exists for 5-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc) 3309751808 bytes in 19.07 secs (173562476 bytes/sec) [Pipeline] { [Pipeline] cache check data successfully Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table foreign_key.finish_mark not exists for 6-th check, retry later wait process cdc.test exit for 1-th time... table charset_gbk_test0.t0 exists table charset_gbk_test0.t1 exists table charset_gbk_test1.t0 exists table test.finish_mark not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 14:57:15 CST 2024] <<<<<< run test case simple success! >>>>>> table test.finish_mark not exists for 2-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table foreign_key.finish_mark not exists for 7-th check, retry later table test.finish_mark not exists for 3-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table foreign_key.finish_mark not exists for 8-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2127b4d80004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f6q5z, pid:6400, start at 2024-05-30 14:57:19.161737283 +0800 CST m=+5.066948918 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:19.168 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:19.158 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:19.158 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2127b4d80004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f6q5z, pid:6400, start at 2024-05-30 14:57:19.161737283 +0800 CST m=+5.066948918 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:19.168 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:19.158 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:19.158 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2127b5880006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f6q5z, pid:6486, start at 2024-05-30 14:57:19.209382451 +0800 CST m=+5.038298626 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:19.216 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:19.202 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:19.202 +0800 All versions after safe point can be accessed. (DO NOT EDIT) table foreign_key.finish_mark not exists for 9-th check, retry later table test.finish_mark not exists for 4-th check, retry later Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/changefeed_pause_resume/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table foreign_key.finish_mark not exists for 10-th check, retry later table test.finish_mark not exists for 5-th check, retry later [Thu May 30 14:57:22 CST 2024] <<<<<< START cdc server in processor_stop_delay case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorStopDelay=1*sleep(10000)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_stop_delay.77407742.out server --log-file /tmp/tidb_cdc_test/processor_stop_delay/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_stop_delay/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc) 3309751808 bytes in 7.46 secs (443500552 bytes/sec) [Pipeline] { [Pipeline] cache =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/changefeed_reconstruct/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table test.finish_mark not exists for 6-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/changefeed_pause_resume Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... table foreign_key.finish_mark not exists for 11-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/cdc_server_tips/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:25 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/60bd1901-b409-489b-8994-dede39437ff9 {"id":"60bd1901-b409-489b-8994-dede39437ff9","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849c77fb4 60bd1901-b409-489b-8994-dede39437ff9 /tidb/cdc/default/default/upstream/7374683178624145344 {"id":7374683178624145344,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/60bd1901-b409-489b-8994-dede39437ff9 {"id":"60bd1901-b409-489b-8994-dede39437ff9","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849c77fb4 60bd1901-b409-489b-8994-dede39437ff9 /tidb/cdc/default/default/upstream/7374683178624145344 {"id":7374683178624145344,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/60bd1901-b409-489b-8994-dede39437ff9 {"id":"60bd1901-b409-489b-8994-dede39437ff9","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc849c77fb4 60bd1901-b409-489b-8994-dede39437ff9 /tidb/cdc/default/default/upstream/7374683178624145344 {"id":7374683178624145344,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Thu May 30 14:57:25 CST 2024] <<<<<< START Pulsar consumer in processor_stop_delay case >>>>>> table test.finish_mark not exists for 7-th check, retry later table processor_stop_delay.t not exists for 1-th check, retry later table foreign_key.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... start tidb cluster in /tmp/tidb_cdc_test/changefeed_reconstruct Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 14:57:27 CST 2024] <<<<<< run test case foreign_key success! >>>>>> table test.finish_mark exists check table exists success table processor_stop_delay.t exists check diff successfully check diff failed 1-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release wait process cdc.test exit for 1-th time... start tidb cluster in /tmp/tidb_cdc_test/cdc_server_tips Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... wait process cdc.test exit for 4-th time... Verifying downstream PD is started... check diff successfully Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 5-th time... process cdc.test already exit [Thu May 30 14:57:30 CST 2024] <<<<<< run test case charset_gbk success! >>>>>> Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release check diff failed 1-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 3-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2128af100013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:4217, start at 2024-05-30 14:57:35.213574495 +0800 CST m=+5.076572360 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:35.221 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:35.222 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:35.222 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2128af100013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:4217, start at 2024-05-30 14:57:35.213574495 +0800 CST m=+5.076572360 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:35.221 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:35.222 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:35.222 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2128b16c0005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:4283, start at 2024-05-30 14:57:35.328241969 +0800 CST m=+5.121445187 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:35.335 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:35.323 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:35.323 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 4-th time, retry later [Thu May 30 14:57:38 CST 2024] <<<<<< START cdc server in changefeed_pause_resume case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_pause_resume.55835585.out server --log-file /tmp/tidb_cdc_test/changefeed_pause_resume/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_pause_resume/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff successfully wait process cdc.test exit for 1-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2128d9740012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5sgr5, pid:4025, start at 2024-05-30 14:57:37.908345292 +0800 CST m=+5.123911812 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:37.915 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:37.885 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:37.885 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2128d9740012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5sgr5, pid:4025, start at 2024-05-30 14:57:37.908345292 +0800 CST m=+5.123911812 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:37.915 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:37.885 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:37.885 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2128dbb00011 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5sgr5, pid:4112, start at 2024-05-30 14:57:38.04892111 +0800 CST m=+5.202227063 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:38.056 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:38.028 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:38.028 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2128eef80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sv1w0, pid:3837, start at 2024-05-30 14:57:39.299350865 +0800 CST m=+5.842795099 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:39.307 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:39.311 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:39.311 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 5-th time, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 14:57:40 CST 2024] <<<<<< run test case kv_client_stream_reconnect success! >>>>>> [Thu May 30 14:57:41 CST 2024] <<<<<< START cdc server in changefeed_reconstruct case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_reconstruct.53555357.out server --log-file /tmp/tidb_cdc_test/changefeed_reconstruct/cdcserver1.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_reconstruct/cdc_dataserver1 --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:41 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c5042b88-b83b-4c1f-8542-1e1158cd3e26 {"id":"c5042b88-b83b-4c1f-8542-1e1158cd3e26","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a0032b7 c5042b88-b83b-4c1f-8542-1e1158cd3e26 /tidb/cdc/default/default/upstream/7374683246568426350 {"id":7374683246568426350,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c5042b88-b83b-4c1f-8542-1e1158cd3e26 {"id":"c5042b88-b83b-4c1f-8542-1e1158cd3e26","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a0032b7 c5042b88-b83b-4c1f-8542-1e1158cd3e26 /tidb/cdc/default/default/upstream/7374683246568426350 {"id":7374683246568426350,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c5042b88-b83b-4c1f-8542-1e1158cd3e26 {"id":"c5042b88-b83b-4c1f-8542-1e1158cd3e26","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a0032b7 c5042b88-b83b-4c1f-8542-1e1158cd3e26 /tidb/cdc/default/default/upstream/7374683246568426350 {"id":7374683246568426350,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2128eef80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sv1w0, pid:3837, start at 2024-05-30 14:57:39.299350865 +0800 CST m=+5.842795099 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:39.307 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:39.311 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:39.311 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2128eba80004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sv1w0, pid:3914, start at 2024-05-30 14:57:39.055217762 +0800 CST m=+5.544050784 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:39.064 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:39.050 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:39.050 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Thu May 30 14:57:41 CST 2024] <<<<<< START Pulsar consumer in changefeed_pause_resume case >>>>>> table changefeed_pause_resume.t1 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/ddl_puller_lag/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/ddl_manager/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... check diff failed 6-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/ddl_puller_lag Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... table sink_retry.finish_mark_1 exists check diff successfully ***************** properties ***************** "mysql.user"="root" "requestdistribution"="uniform" "mysql.port"="4000" "mysql.db"="sink_retry" "recordcount"="10" "readallfields"="true" "threadcount"="2" "mysql.host"="127.0.0.1" "readproportion"="0" "dotransactions"="false" "operationcount"="0" "scanproportion"="0" "updateproportion"="0" "insertproportion"="0" "workload"="core" ********************************************** Run finished, takes 5.616533ms INSERT - Takes(s): 0.0, Count: 10, OPS: 3081.1, Avg(us): 988, Min(us): 552, Max(us): 2548, 95th(us): 3000, 99th(us): 3000 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.cli.5156.out cli tso query --pd=http://127.0.0.1:2379 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:44 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/af11961b-22ea-423f-8f99-32b6658ffb45 {"id":"af11961b-22ea-423f-8f99-32b6658ffb45","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a0a97bd af11961b-22ea-423f-8f99-32b6658ffb45 /tidb/cdc/default/default/upstream/7374683258741615610 {"id":7374683258741615610,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/af11961b-22ea-423f-8f99-32b6658ffb45 {"id":"af11961b-22ea-423f-8f99-32b6658ffb45","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a0a97bd af11961b-22ea-423f-8f99-32b6658ffb45 /tidb/cdc/default/default/upstream/7374683258741615610 {"id":7374683258741615610,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/af11961b-22ea-423f-8f99-32b6658ffb45 {"id":"af11961b-22ea-423f-8f99-32b6658ffb45","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a0a97bd af11961b-22ea-423f-8f99-32b6658ffb45 /tidb/cdc/default/default/upstream/7374683258741615610 {"id":7374683258741615610,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Thu May 30 14:57:44 CST 2024] <<<<<< START Pulsar consumer in changefeed_reconstruct case >>>>>> table changefeed_pause_resume.t1 exists table changefeed_pause_resume.t2 not exists for 1-th check, retry later check diff successfully ***************** properties ***************** "mysql.db"="changefeed_reconstruct" "workload"="core" "requestdistribution"="uniform" "mysql.user"="root" "readproportion"="0" "insertproportion"="0" "dotransactions"="false" "readallfields"="true" "recordcount"="50" "mysql.host"="127.0.0.1" "updateproportion"="0" "mysql.port"="4000" "threadcount"="4" "operationcount"="0" "scanproportion"="0" ********************************************** Run finished, takes 21.995162ms INSERT - Takes(s): 0.0, Count: 48, OPS: 2828.2, Avg(us): 1660, Min(us): 978, Max(us): 5261, 95th(us): 5000, 99th(us): 6000 table changefeed_reconstruct.usertable not exists for 1-th check, retry later wait process cdc.test exit for 1-th time... start tidb cluster in /tmp/tidb_cdc_test/ddl_manager Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release wait process cdc.test exit for 2-th time... + set +x + tso='450114948723113988 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114948723113988 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x try a VALID cdc server command [Thu May 30 14:57:45 CST 2024] <<<<<< START cdc server in cdc_server_tips case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.51985200.out server --log-file /tmp/tidb_cdc_test/cdc_server_tips/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cdc_server_tips/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 14:57:46 CST 2024] <<<<<< run test case processor_stop_delay success! >>>>>> table changefeed_pause_resume.t2 exists table changefeed_pause_resume.t3 not exists for 1-th check, retry later table changefeed_reconstruct.usertable exists check diff failed 1-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table changefeed_pause_resume.t3 exists Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:48 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/03577e0e-16ee-4d44-82b4-b889667949ba {"id":"03577e0e-16ee-4d44-82b4-b889667949ba","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a10f0d0 03577e0e-16ee-4d44-82b4-b889667949ba /tidb/cdc/default/default/upstream/7374683255463190839 {"id":7374683255463190839,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/03577e0e-16ee-4d44-82b4-b889667949ba {"id":"03577e0e-16ee-4d44-82b4-b889667949ba","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a10f0d0 03577e0e-16ee-4d44-82b4-b889667949ba /tidb/cdc/default/default/upstream/7374683255463190839 {"id":7374683255463190839,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/03577e0e-16ee-4d44-82b4-b889667949ba {"id":"03577e0e-16ee-4d44-82b4-b889667949ba","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a10f0d0 03577e0e-16ee-4d44-82b4-b889667949ba /tidb/cdc/default/default/upstream/7374683255463190839 {"id":7374683255463190839,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check diff successfully wait process 5360 exit for 1-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process 5360 exit for 2-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/cdc/run.sh using Sink-Type: pulsar... <<================= check diff failed 1-th time, retry later /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (5360) - No such process wait process 5360 exit for 3-th time... process 5360 already exit check_no_capture http://127.0.0.1:2379 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Thu May 30 14:57:50 CST 2024] <<<<<< START cdc server in changefeed_reconstruct case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_reconstruct.56605662.out server --log-file /tmp/tidb_cdc_test/changefeed_reconstruct/cdcserver2.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_reconstruct/cdc_dataserver2 --cluster-id default --addr 127.0.0.1:8300 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2129bf40000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-mwr1g, pid:3839, start at 2024-05-30 14:57:52.60309805 +0800 CST m=+5.167366004 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:52.609 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:52.592 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:52.592 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2129bf40000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-mwr1g, pid:3839, start at 2024-05-30 14:57:52.60309805 +0800 CST m=+5.167366004 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:52.609 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:52.592 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:52.592 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2129bff40017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-mwr1g, pid:3910, start at 2024-05-30 14:57:52.669883505 +0800 CST m=+5.184798477 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:52.677 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:52.637 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:52.637 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:57:53 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/6a0ea766-ace0-4dd5-96b2-a071fa7eedf6 {UpstreamID:7374683258741615610 Namespace:default ID:6a0ea766-ace0-4dd5-96b2-a071fa7eedf6 SinkURI:pulsar://127.0.0.1:6650/ticdc-changefeed-reconstruct-27398?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 14:57:44.479261926 +0800 CST StartTs:450114948768727045 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004888120 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114948808310790} {CheckpointTs:450114951010058248 MinTableBarrierTs:450114951010058248 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114951010058248, checkpointTs: 450114951010058248, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dccf8648-5086-4ee7-83be-c591aadd5b31 {"id":"dccf8648-5086-4ee7-83be-c591aadd5b31","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a0a9846 dccf8648-5086-4ee7-83be-c591aadd5b31 /tidb/cdc/default/default/changefeed/info/6a0ea766-ace0-4dd5-96b2-a071fa7eedf6 {"upstream-id":7374683258741615610,"namespace":"default","changefeed-id":"6a0ea766-ace0-4dd5-96b2-a071fa7eedf6","sink-uri":"pulsar://127.0.0.1:6650/ticdc-changefeed-reconstruct-27398?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T14:57:44.479261926+08:00","start-ts":450114948768727045,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114948808310790} /tidb/cdc/default/default/changefeed/status/6a0ea766-ace0-4dd5-96b2-a071fa7eedf6 {"checkpoint-ts":450114951010058248,"min-table-barrier-ts":450114951010058248,"admin-job-type":0} /tidb/cdc/default/default/task/position/dccf8648-5086-4ee7-83be-c591aadd5b31/6a0ea766-ace0-4dd5-96b2-a071fa7eedf6 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683258741615610 {"id":7374683258741615610,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/6a0ea766-ace0-4dd5-96b2-a071fa7eedf6 {UpstreamID:7374683258741615610 Namespace:default ID:6a0ea766-ace0-4dd5-96b2-a071fa7eedf6 SinkURI:pulsar://127.0.0.1:6650/ticdc-changefeed-reconstruct-27398?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 14:57:44.479261926 +0800 CST StartTs:450114948768727045 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004888120 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114948808310790} {CheckpointTs:450114951010058248 MinTableBarrierTs:450114951010058248 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114951010058248, checkpointTs: 450114951010058248, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dccf8648-5086-4ee7-83be-c591aadd5b31 {"id":"dccf8648-5086-4ee7-83be-c591aadd5b31","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a0a9846 dccf8648-5086-4ee7-83be-c591aadd5b31 /tidb/cdc/default/default/changefeed/info/6a0ea766-ace0-4dd5-96b2-a071fa7eedf6 {"upstream-id":7374683258741615610,"namespace":"default","changefeed-id":"6a0ea766-ace0-4dd5-96b2-a071fa7eedf6","sink-uri":"pulsar://127.0.0.1:6650/ticdc-changefeed-reconstruct-27398?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T14:57:44.479261926+08:00","start-ts":450114948768727045,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114948808310790} /tidb/cdc/default/default/changefeed/status/6a0ea766-ace0-4dd5-96b2-a071fa7eedf6 {"checkpoint-ts":450114951010058248,"min-table-barrier-ts":450114951010058248,"admin-job-type":0} /tidb/cdc/default/default/task/position/dccf8648-5086-4ee7-83be-c591aadd5b31/6a0ea766-ace0-4dd5-96b2-a071fa7eedf6 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683258741615610 {"id":7374683258741615610,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/6a0ea766-ace0-4dd5-96b2-a071fa7eedf6 {UpstreamID:7374683258741615610 Namespace:default ID:6a0ea766-ace0-4dd5-96b2-a071fa7eedf6 SinkURI:pulsar://127.0.0.1:6650/ticdc-changefeed-reconstruct-27398?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 14:57:44.479261926 +0800 CST StartTs:450114948768727045 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004888120 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114948808310790} {CheckpointTs:450114951010058248 MinTableBarrierTs:450114951010058248 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450114951010058248, checkpointTs: 450114951010058248, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dccf8648-5086-4ee7-83be-c591aadd5b31 {"id":"dccf8648-5086-4ee7-83be-c591aadd5b31","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a0a9846 dccf8648-5086-4ee7-83be-c591aadd5b31 /tidb/cdc/default/default/changefeed/info/6a0ea766-ace0-4dd5-96b2-a071fa7eedf6 {"upstream-id":7374683258741615610,"namespace":"default","changefeed-id":"6a0ea766-ace0-4dd5-96b2-a071fa7eedf6","sink-uri":"pulsar://127.0.0.1:6650/ticdc-changefeed-reconstruct-27398?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T14:57:44.479261926+08:00","start-ts":450114948768727045,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114948808310790} /tidb/cdc/default/default/changefeed/status/6a0ea766-ace0-4dd5-96b2-a071fa7eedf6 {"checkpoint-ts":450114951010058248,"min-table-barrier-ts":450114951010058248,"admin-job-type":0} /tidb/cdc/default/default/task/position/dccf8648-5086-4ee7-83be-c591aadd5b31/6a0ea766-ace0-4dd5-96b2-a071fa7eedf6 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683258741615610 {"id":7374683258741615610,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list --pd=http://127.0.0.1:2379 2>&1 | grep id check diff failed 3-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2129d9dc000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-blflz, pid:3758, start at 2024-05-30 14:57:54.310255882 +0800 CST m=+5.217721147 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:54.318 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:54.295 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:54.295 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2129d9dc000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-blflz, pid:3758, start at 2024-05-30 14:57:54.310255882 +0800 CST m=+5.217721147 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:54.318 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:54.295 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:54.295 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2129db600013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-blflz, pid:3841, start at 2024-05-30 14:57:54.420223916 +0800 CST m=+5.265361785 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-14:59:54.428 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:57:54.392 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:47:54.392 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_manager/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_manager/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } "id": "dccf8648-5086-4ee7-83be-c591aadd5b31", "cluster-id": "default" run task successfully =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/partition_table/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.cli.5160.out cli tso query --pd=http://127.0.0.1:2379 capture_id: dccf8648-5086-4ee7-83be-c591aadd5b31 check_processor_table_count http://127.0.0.1:2379 6a0ea766-ace0-4dd5-96b2-a071fa7eedf6 dccf8648-5086-4ee7-83be-c591aadd5b31 1 check diff failed 4-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/split_region/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... run task successfully check_processor_table_count http://127.0.0.1:2379 6a0ea766-ace0-4dd5-96b2-a071fa7eedf6 dccf8648-5086-4ee7-83be-c591aadd5b31 0 table count 1 does equal to expected count 0 run task failed 1-th time, retry later + set +x + tso='450114951850754049 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114951850754049 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 14:57:57 CST 2024] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.51445146.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Thu May 30 14:57:58 CST 2024] <<<<<< START cdc server in ddl_puller_lag case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorDDLResolved=1*sleep(180000)' + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.52125214.out server --log-file /tmp/tidb_cdc_test/ddl_puller_lag/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_puller_lag/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 start tidb cluster in /tmp/tidb_cdc_test/partition_table Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... check diff failed 5-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/split_region Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... check_processor_table_count http://127.0.0.1:2379 6a0ea766-ace0-4dd5-96b2-a071fa7eedf6 dccf8648-5086-4ee7-83be-c591aadd5b31 0 run task successfully ***************** properties ***************** "insertproportion"="0" "operationcount"="0" "mysql.port"="4000" "dotransactions"="false" "readproportion"="0" "mysql.user"="root" "readallfields"="true" "workload"="core" "recordcount"="50" "threadcount"="4" "mysql.db"="changefeed_reconstruct" "updateproportion"="0" "mysql.host"="127.0.0.1" "scanproportion"="0" "requestdistribution"="uniform" ********************************************** Run finished, takes 16.116896ms INSERT - Takes(s): 0.0, Count: 48, OPS: 3902.1, Avg(us): 1257, Min(us): 825, Max(us): 4064, 95th(us): 4000, 99th(us): 5000 table changefeed_reconstruct.usertable not exists for 1-th check, retry later check diff successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:00 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e451a8a7-2083-41ed-a1d7-154981ad4f5c {"id":"e451a8a7-2083-41ed-a1d7-154981ad4f5c","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a521db7 e451a8a7-2083-41ed-a1d7-154981ad4f5c /tidb/cdc/default/default/upstream/7374683327509655426 {"id":7374683327509655426,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e451a8a7-2083-41ed-a1d7-154981ad4f5c {"id":"e451a8a7-2083-41ed-a1d7-154981ad4f5c","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a521db7 e451a8a7-2083-41ed-a1d7-154981ad4f5c /tidb/cdc/default/default/upstream/7374683327509655426 {"id":7374683327509655426,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e451a8a7-2083-41ed-a1d7-154981ad4f5c {"id":"e451a8a7-2083-41ed-a1d7-154981ad4f5c","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a521db7 e451a8a7-2083-41ed-a1d7-154981ad4f5c /tidb/cdc/default/default/upstream/7374683327509655426 {"id":7374683327509655426,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.cli.5194.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-25055?protocol=canal-json&enable-tidb-extension=true' -c=ddl-manager Create changefeed successfully! ID: ddl-manager Info: {"upstream_id":7374683327509655426,"namespace":"default","id":"ddl-manager","sink_uri":"pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-25055?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T14:58:01.105345244+08:00","start_ts":450114953123201026,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114953123201026,"checkpoint_ts":450114953123201026,"checkpoint_time":"2024-05-30 14:58:00.896"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... Verifying downstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:01 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5672a3cf-d471-455a-a8d2-8135da534c07 {"id":"5672a3cf-d471-455a-a8d2-8135da534c07","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a4baadd 5672a3cf-d471-455a-a8d2-8135da534c07 /tidb/cdc/default/default/upstream/7374683322238406210 {"id":7374683322238406210,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5672a3cf-d471-455a-a8d2-8135da534c07 {"id":"5672a3cf-d471-455a-a8d2-8135da534c07","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a4baadd 5672a3cf-d471-455a-a8d2-8135da534c07 /tidb/cdc/default/default/upstream/7374683322238406210 {"id":7374683322238406210,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5672a3cf-d471-455a-a8d2-8135da534c07 {"id":"5672a3cf-d471-455a-a8d2-8135da534c07","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a4baadd 5672a3cf-d471-455a-a8d2-8135da534c07 /tidb/cdc/default/default/upstream/7374683322238406210 {"id":7374683322238406210,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.cli.5263.out cli changefeed create --start-ts=450114951850754049 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-ddl-puller-lag-test-28642?protocol=canal-json&enable-tidb-extension=true' table changefeed_reconstruct.usertable exists check diff failed 1-th time, retry later Create changefeed successfully! ID: 98819e1f-cfee-437c-b58b-6fef2aaabb0e Info: {"upstream_id":7374683322238406210,"namespace":"default","id":"98819e1f-cfee-437c-b58b-6fef2aaabb0e","sink_uri":"pulsar://127.0.0.1:6650/ticdc-ddl-puller-lag-test-28642?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T14:58:02.086170245+08:00","start_ts":450114951850754049,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114951850754049,"checkpoint_ts":450114951850754049,"checkpoint_time":"2024-05-30 14:57:56.042"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... check diff failed 1-th time, retry later + set +x [Thu May 30 14:58:02 CST 2024] <<<<<< START Pulsar consumer in ddl_manager case >>>>>> Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + set +x [Thu May 30 14:58:03 CST 2024] <<<<<< START Pulsar consumer in ddl_puller_lag case >>>>>> check diff successfully wait process cdc.test exit for 1-th time... Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc) 3309751808 bytes in 39.45 secs (83892461 bytes/sec) [Pipeline] { [Pipeline] cache wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 14:58:05 CST 2024] <<<<<< run test case changefeed_reconstruct success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process 5149 exit for 1-th time... wait process 5149 exit for 2-th time... check diff failed 3-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process 5149 exit for 3-th time... wait process 5149 exit for 4-th time... wait process 5149 exit for 5-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process 5149 exit for 6-th time... check diff failed 4-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process 5149 exit for 7-th time... wait process 5149 exit for 8-th time... wait process 5149 exit for 9-th time... wait process 5149 exit for 10-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212ad1680013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5pr5v, pid:4056, start at 2024-05-30 14:58:10.165584457 +0800 CST m=+5.537452745 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:10.175 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:10.138 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:10.138 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212ac76c0017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f6q5z, pid:9223, start at 2024-05-30 14:58:09.540077395 +0800 CST m=+5.102498085 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:09.547 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:09.549 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:09.549 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212ac76c0017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f6q5z, pid:9223, start at 2024-05-30 14:58:09.540077395 +0800 CST m=+5.102498085 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:09.547 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:09.549 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:09.549 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212ac7640015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-f6q5z, pid:9302, start at 2024-05-30 14:58:09.53982913 +0800 CST m=+5.052180232 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:09.547 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:09.548 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:09.548 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/partition_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/partition_table/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } valid ~~~ running cdc Failed to start cdc, the usage tips should be printed 1st test case cdc_server_tips success! try an INVALID cdc server command [Thu May 30 14:58:08 CST 2024] <<<<<< START cdc server in cdc_server_tips case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ true != \n\o ]] + GO_FAILPOINTS= + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.52745276.out server --log-file /tmp/tidb_cdc_test/cdc_server_tips/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cdc_server_tips/cdc_data --cluster-id default --pd None wait process 5149 exit for 11-th time... check diff failed 5-th time, retry later /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (5149) - No such process wait process 5149 exit for 12-th time... process 5149 already exit [Thu May 30 14:58:11 CST 2024] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.53095311.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212ad1680013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5pr5v, pid:4056, start at 2024-05-30 14:58:10.165584457 +0800 CST m=+5.537452745 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:10.175 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:10.138 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:10.138 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212ad4740014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5pr5v, pid:4140, start at 2024-05-30 14:58:10.356980523 +0800 CST m=+5.640192546 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:10.366 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:10.333 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:10.333 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc) 3309751808 bytes in 5.42 secs (610814102 bytes/sec) [Pipeline] { [Pipeline] cache + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.cli.10594.out cli tso query --pd=http://127.0.0.1:2379 Logging trace to /tmp/tidb_cdc_test/split_region/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/split_region/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/split_region/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/split_region/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/split_region/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 6-th time, retry later + set +x + tso='450114956309037057 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114956309037057 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 14:58:14 CST 2024] <<<<<< START cdc server in partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.1063110633.out server --log-file /tmp/tidb_cdc_test/partition_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/partition_table/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.cli.5448.out cli tso query --pd=http://127.0.0.1:2379 check diff successfully table sink_retry.finish_mark_2 exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 14:58:14 CST 2024] <<<<<< run test case sink_retry success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/multi_capture/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... + set +x + tso='450114956895977476 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114956895977476 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x check diff failed 1-th time, retry later [Thu May 30 14:58:17 CST 2024] <<<<<< START cdc server in split_region case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.54815483.out server --log-file /tmp/tidb_cdc_test/split_region/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/split_region/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:16 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7374683327509655426 Namespace:default ID:ddl-manager SinkURI:pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-25055?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 14:58:01.105345244 +0800 CST StartTs:450114953123201026 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037783f0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114953175629826} {CheckpointTs:450114953555738656 MinTableBarrierTs:450114953555738656 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a60558ef-1e45-420b-b1ba-41f9fb3f8512 {"id":"a60558ef-1e45-420b-b1ba-41f9fb3f8512","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a521ff2 a60558ef-1e45-420b-b1ba-41f9fb3f8512 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7374683327509655426,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-25055?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T14:58:01.105345244+08:00","start-ts":450114953123201026,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114953175629826} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":450114953555738656,"min-table-barrier-ts":450114953555738656,"admin-job-type":0} /tidb/cdc/default/default/task/position/a60558ef-1e45-420b-b1ba-41f9fb3f8512/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683327509655426 {"id":7374683327509655426,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7374683327509655426 Namespace:default ID:ddl-manager SinkURI:pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-25055?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 14:58:01.105345244 +0800 CST StartTs:450114953123201026 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037783f0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114953175629826} {CheckpointTs:450114953555738656 MinTableBarrierTs:450114953555738656 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a60558ef-1e45-420b-b1ba-41f9fb3f8512 {"id":"a60558ef-1e45-420b-b1ba-41f9fb3f8512","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a521ff2 a60558ef-1e45-420b-b1ba-41f9fb3f8512 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7374683327509655426,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-25055?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T14:58:01.105345244+08:00","start-ts":450114953123201026,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114953175629826} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":450114953555738656,"min-table-barrier-ts":450114953555738656,"admin-job-type":0} /tidb/cdc/default/default/task/position/a60558ef-1e45-420b-b1ba-41f9fb3f8512/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683327509655426 {"id":7374683327509655426,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7374683327509655426 Namespace:default ID:ddl-manager SinkURI:pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-25055?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 14:58:01.105345244 +0800 CST StartTs:450114953123201026 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037783f0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114953175629826} {CheckpointTs:450114953555738656 MinTableBarrierTs:450114953555738656 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a60558ef-1e45-420b-b1ba-41f9fb3f8512 {"id":"a60558ef-1e45-420b-b1ba-41f9fb3f8512","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a521ff2 a60558ef-1e45-420b-b1ba-41f9fb3f8512 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7374683327509655426,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-25055?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T14:58:01.105345244+08:00","start-ts":450114953123201026,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114953175629826} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":450114953555738656,"min-table-barrier-ts":450114953555738656,"admin-job-type":0} /tidb/cdc/default/default/task/position/a60558ef-1e45-420b-b1ba-41f9fb3f8512/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683327509655426 {"id":7374683327509655426,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Thu May 30 14:58:16 CST 2024] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.53715373.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:17 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/12d79870-611b-4bf3-b195-a219a5ac5cd2 {"id":"12d79870-611b-4bf3-b195-a219a5ac5cd2","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a8649be 12d79870-611b-4bf3-b195-a219a5ac5cd2 /tidb/cdc/default/default/upstream/7374683393746972927 {"id":7374683393746972927,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/12d79870-611b-4bf3-b195-a219a5ac5cd2 {"id":"12d79870-611b-4bf3-b195-a219a5ac5cd2","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a8649be 12d79870-611b-4bf3-b195-a219a5ac5cd2 /tidb/cdc/default/default/upstream/7374683393746972927 {"id":7374683393746972927,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/12d79870-611b-4bf3-b195-a219a5ac5cd2 {"id":"12d79870-611b-4bf3-b195-a219a5ac5cd2","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a8649be 12d79870-611b-4bf3-b195-a219a5ac5cd2 /tidb/cdc/default/default/upstream/7374683393746972927 {"id":7374683393746972927,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.cli.10681.out cli changefeed create --start-ts=450114956309037057 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-partition-table-test-17006?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: 24f20fff-aa93-4bb5-ae7b-00012f91f1f5 Info: {"upstream_id":7374683393746972927,"namespace":"default","id":"24f20fff-aa93-4bb5-ae7b-00012f91f1f5","sink_uri":"pulsar://127.0.0.1:6650/ticdc-partition-table-test-17006?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T14:58:18.056503792+08:00","start_ts":450114956309037057,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114956309037057,"checkpoint_ts":450114956309037057,"checkpoint_time":"2024-05-30 14:58:13.049"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... check diff failed 2-th time, retry later + set +x [Thu May 30 14:58:19 CST 2024] <<<<<< START Pulsar consumer in partition_table case >>>>>> < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:19 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7374683327509655426 Namespace:default ID:ddl-manager SinkURI:pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-25055?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 14:58:01.105345244 +0800 CST StartTs:450114953123201026 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037783f0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114953175629826} {CheckpointTs:450114953555738656 MinTableBarrierTs:450114953555738656 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a60558ef-1e45-420b-b1ba-41f9fb3f8512 {"id":"a60558ef-1e45-420b-b1ba-41f9fb3f8512","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a521ff2 a60558ef-1e45-420b-b1ba-41f9fb3f8512 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7374683327509655426,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-25055?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T14:58:01.105345244+08:00","start-ts":450114953123201026,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114953175629826} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":450114953581953058,"min-table-barrier-ts":450114953581953058,"admin-job-type":0} /tidb/cdc/default/default/task/position/a60558ef-1e45-420b-b1ba-41f9fb3f8512/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683327509655426 {"id":7374683327509655426,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7374683327509655426 Namespace:default ID:ddl-manager SinkURI:pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-25055?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 14:58:01.105345244 +0800 CST StartTs:450114953123201026 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037783f0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114953175629826} {CheckpointTs:450114953555738656 MinTableBarrierTs:450114953555738656 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a60558ef-1e45-420b-b1ba-41f9fb3f8512 {"id":"a60558ef-1e45-420b-b1ba-41f9fb3f8512","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a521ff2 a60558ef-1e45-420b-b1ba-41f9fb3f8512 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7374683327509655426,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-25055?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T14:58:01.105345244+08:00","start-ts":450114953123201026,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114953175629826} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":450114953581953058,"min-table-barrier-ts":450114953581953058,"admin-job-type":0} /tidb/cdc/default/default/task/position/a60558ef-1e45-420b-b1ba-41f9fb3f8512/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683327509655426 {"id":7374683327509655426,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7374683327509655426 Namespace:default ID:ddl-manager SinkURI:pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-25055?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 14:58:01.105345244 +0800 CST StartTs:450114953123201026 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037783f0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114953175629826} {CheckpointTs:450114953555738656 MinTableBarrierTs:450114953555738656 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a60558ef-1e45-420b-b1ba-41f9fb3f8512 {"id":"a60558ef-1e45-420b-b1ba-41f9fb3f8512","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a521ff2 a60558ef-1e45-420b-b1ba-41f9fb3f8512 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7374683327509655426,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-25055?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T14:58:01.105345244+08:00","start-ts":450114953123201026,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114953175629826} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":450114953581953058,"min-table-barrier-ts":450114953581953058,"admin-job-type":0} /tidb/cdc/default/default/task/position/a60558ef-1e45-420b-b1ba-41f9fb3f8512/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683327509655426 {"id":7374683327509655426,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table ddl_manager.finish_mark not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:20 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/51cf277e-fa71-42c8-9c6e-d8ea352cd901 {"id":"51cf277e-fa71-42c8-9c6e-d8ea352cd901","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a87fbe9 51cf277e-fa71-42c8-9c6e-d8ea352cd901 /tidb/cdc/default/default/upstream/7374683391158173938 {"id":7374683391158173938,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/51cf277e-fa71-42c8-9c6e-d8ea352cd901 {"id":"51cf277e-fa71-42c8-9c6e-d8ea352cd901","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a87fbe9 51cf277e-fa71-42c8-9c6e-d8ea352cd901 /tidb/cdc/default/default/upstream/7374683391158173938 {"id":7374683391158173938,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/51cf277e-fa71-42c8-9c6e-d8ea352cd901 {"id":"51cf277e-fa71-42c8-9c6e-d8ea352cd901","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84a87fbe9 51cf277e-fa71-42c8-9c6e-d8ea352cd901 /tidb/cdc/default/default/upstream/7374683391158173938 {"id":7374683391158173938,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.cli.5530.out cli changefeed create --start-ts=450114956895977476 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-split-region-test-10531?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: e80e57f7-8375-4599-a15d-ea341e70d752 Info: {"upstream_id":7374683391158173938,"namespace":"default","id":"e80e57f7-8375-4599-a15d-ea341e70d752","sink_uri":"pulsar://127.0.0.1:6650/ticdc-split-region-test-10531?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T14:58:20.569466786+08:00","start_ts":450114956895977476,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114956895977476,"checkpoint_ts":450114956895977476,"checkpoint_time":"2024-05-30 14:58:15.288"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... check diff failed 3-th time, retry later + set +x [Thu May 30 14:58:22 CST 2024] <<<<<< START Pulsar consumer in split_region case >>>>>> table split_region.test1 not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 2-th check, retry later check diff failed 4-th time, retry later table ddl_manager.finish_mark not exists for 3-th check, retry later table split_region.test1 exists table split_region.test2 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/changefeed_error/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/multi_capture Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... check diff failed 5-th time, retry later table ddl_manager.finish_mark not exists for 4-th check, retry later table split_region.test2 exists check diff failed 1-th time, retry later Verifying downstream PD is started... check diff successfully start tidb cluster in /tmp/tidb_cdc_test/changefeed_error Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... table ddl_manager.finish_mark not exists for 5-th check, retry later check diff successfully Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release invalid ~~~ running cdc Failed to start cdc, the usage tips should be printed 2nd test case cdc_server_tips success! [Thu May 30 14:58:28 CST 2024] <<<<<< run all test cases cdc_server_tips success! >>>>>> table partition_table.t exists table partition_table.t1 exists table partition_table.t2 not exists for 1-th check, retry later check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 6-th check, retry later table partition_table.t2 not exists for 2-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 7-th check, retry later table partition_table.t2 not exists for 3-th check, retry later check diff failed 3-th time, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 8-th check, retry later table partition_table.t2 not exists for 4-th check, retry later check diff failed 4-th time, retry later table ddl_manager.finish_mark not exists for 9-th check, retry later table partition_table.t2 exists table partition_table.finish_mark not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212c56c0000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5sgr5, pid:6977, start at 2024-05-30 14:58:35.067309385 +0800 CST m=+4.989525685 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:35.072 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:35.056 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:35.056 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212c56c0000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5sgr5, pid:6977, start at 2024-05-30 14:58:35.067309385 +0800 CST m=+4.989525685 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:35.072 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:35.056 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:35.056 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212c58400015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-5sgr5, pid:7047, start at 2024-05-30 14:58:35.198741516 +0800 CST m=+5.054574505 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:35.205 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:35.202 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:35.202 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_capture/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_capture/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc) 3309751808 bytes in 21.31 secs (155321407 bytes/sec) [Pipeline] { [Pipeline] cache check diff successfully table ddl_manager.finish_mark not exists for 10-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.cli.8296.out cli tso query --pd=http://127.0.0.1:2379 table partition_table.finish_mark not exists for 2-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 11-th check, retry later + set +x + tso='450114963034865665 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114963034865665 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ***************** properties ***************** "requestdistribution"="uniform" "mysql.user"="root" "workload"="core" "mysql.port"="4000" "mysql.db"="multi_capture_1" "updateproportion"="0" "insertproportion"="0" "recordcount"="10" "scanproportion"="0" "threadcount"="2" "operationcount"="0" "readproportion"="0" "dotransactions"="false" "mysql.host"="127.0.0.1" "readallfields"="true" ********************************************** Run finished, takes 8.415478ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2141.2, Avg(us): 1592, Min(us): 903, Max(us): 3719, 95th(us): 4000, 99th(us): 4000 table partition_table.finish_mark not exists for 3-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212ca3fc0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-rbkd5, pid:3837, start at 2024-05-30 14:58:40.033705844 +0800 CST m=+5.976764469 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:40.044 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:40.048 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:40.048 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212ca3fc0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-rbkd5, pid:3837, start at 2024-05-30 14:58:40.033705844 +0800 CST m=+5.976764469 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:40.044 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:40.048 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:40.048 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212ca3f00015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-rbkd5, pid:3912, start at 2024-05-30 14:58:40.04043603 +0800 CST m=+5.894653080 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:40.049 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:40.046 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:40.046 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ***************** properties ***************** "insertproportion"="0" "workload"="core" "readproportion"="0" "updateproportion"="0" "recordcount"="10" "threadcount"="2" "readallfields"="true" "mysql.host"="127.0.0.1" "dotransactions"="false" "scanproportion"="0" "operationcount"="0" "mysql.port"="4000" "mysql.user"="root" "mysql.db"="multi_capture_2" "requestdistribution"="uniform" ********************************************** Run finished, takes 7.831003ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2502.7, Avg(us): 1496, Min(us): 868, Max(us): 3728, 95th(us): 4000, 99th(us): 4000 ***************** properties ***************** "scanproportion"="0" "insertproportion"="0" "mysql.port"="4000" "requestdistribution"="uniform" "recordcount"="10" "dotransactions"="false" "updateproportion"="0" "mysql.db"="multi_capture_3" "readallfields"="true" "readproportion"="0" "mysql.host"="127.0.0.1" "mysql.user"="root" "operationcount"="0" "threadcount"="2" "workload"="core" ********************************************** Run finished, takes 7.9223ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2410.9, Avg(us): 1502, Min(us): 795, Max(us): 3659, 95th(us): 4000, 99th(us): 4000 Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_error/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_error/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ***************** properties ***************** "threadcount"="2" "operationcount"="0" "readallfields"="true" "insertproportion"="0" "requestdistribution"="uniform" "mysql.user"="root" "mysql.host"="127.0.0.1" "mysql.port"="4000" "readproportion"="0" "scanproportion"="0" "dotransactions"="false" "workload"="core" "recordcount"="10" "updateproportion"="0" "mysql.db"="multi_capture_4" ********************************************** Run finished, takes 7.855474ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2368.9, Avg(us): 1484, Min(us): 872, Max(us): 3580, 95th(us): 4000, 99th(us): 4000 [Thu May 30 14:58:40 CST 2024] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.84088410.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/event_filter/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... check diff failed 2-th time, retry later table ddl_manager.finish_mark not exists for 12-th check, retry later table partition_table.finish_mark not exists for 4-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/event_filter Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5128.out cli tso query --pd=http://127.0.0.1:2379 check diff failed 3-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:44 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/44248fdc-6ce8-453e-bd9d-20a08f2e184c {"id":"44248fdc-6ce8-453e-bd9d-20a08f2e184c","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84aec4325 44248fdc-6ce8-453e-bd9d-20a08f2e184c /tidb/cdc/default/default/upstream/7374683499012896468 {"id":7374683499012896468,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/44248fdc-6ce8-453e-bd9d-20a08f2e184c {"id":"44248fdc-6ce8-453e-bd9d-20a08f2e184c","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84aec4325 44248fdc-6ce8-453e-bd9d-20a08f2e184c /tidb/cdc/default/default/upstream/7374683499012896468 {"id":7374683499012896468,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/44248fdc-6ce8-453e-bd9d-20a08f2e184c {"id":"44248fdc-6ce8-453e-bd9d-20a08f2e184c","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84aec4325 44248fdc-6ce8-453e-bd9d-20a08f2e184c /tidb/cdc/default/default/upstream/7374683499012896468 {"id":7374683499012896468,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Thu May 30 14:58:44 CST 2024] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.84588460.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 13-th check, retry later table partition_table.finish_mark not exists for 5-th check, retry later check diff failed 4-th time, retry later + set +x + tso='450114964225523713 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114964225523713 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ***************** properties ***************** "operationcount"="0" "threadcount"="4" "workload"="core" "readproportion"="0" "requestdistribution"="uniform" "mysql.user"="root" "mysql.db"="changefeed_error" "mysql.host"="127.0.0.1" "dotransactions"="false" "recordcount"="20" "updateproportion"="0" "scanproportion"="0" "insertproportion"="0" "readallfields"="true" "mysql.port"="4000" ********************************************** Run finished, takes 10.33908ms INSERT - Takes(s): 0.0, Count: 18, OPS: 3323.2, Avg(us): 1620, Min(us): 918, Max(us): 4738, 95th(us): 5000, 99th(us): 5000 [Thu May 30 14:58:45 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/NewChangefeedNoRetryError=1*return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.51865188.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 14-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc) 3309751808 bytes in 6.68 secs (495741014 bytes/sec) [Pipeline] { table partition_table.finish_mark not exists for 6-th check, retry later [Pipeline] sh [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G16 Run cases: owner_resign processor_etcd_worker_delay sink_hang PROW_JOB_ID=f6375c57-288c-4d36-bd8b-6f8b308ffae9 JENKINS_NODE_COOKIE=19e61584-53a3-48e0-b48c-44265c07d314 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/ GOLANG_VERSION=1.21.10 HOSTNAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-k5r94 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test","buildid":"1796069089735610368","prowjobid":"f6375c57-288c-4d36-bd8b-6f8b308ffae9","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#462 TEST_GROUP=G16 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089735610368 GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-z0mtv GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-3335c3054d39d880f4ab7664857409e24c98a0c490a94485ce4d2e2d0717ce44 NODE_LABELS=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-z0mtv p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-k5r94 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-k5r94 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=462 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/owner_resign/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:47 GMT < Content-Length: 867 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/44248fdc-6ce8-453e-bd9d-20a08f2e184c {"id":"44248fdc-6ce8-453e-bd9d-20a08f2e184c","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/ac0bdbd1-c7de-44d0-b033-a5927bdb4a45 {"id":"ac0bdbd1-c7de-44d0-b033-a5927bdb4a45","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84aec4325 44248fdc-6ce8-453e-bd9d-20a08f2e184c /tidb/cdc/default/__cdc_meta__/owner/22318fc84aec4352 ac0bdbd1-c7de-44d0-b033-a5927bdb4a45 /tidb/cdc/default/default/upstream/7374683499012896468 {"id":7374683499012896468,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/44248fdc-6ce8-453e-bd9d-20a08f2e184c {"id":"44248fdc-6ce8-453e-bd9d-20a08f2e184c","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/ac0bdbd1-c7de-44d0-b033-a5927bdb4a45 {"id":"ac0bdbd1-c7de-44d0-b033-a5927bdb4a45","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84aec4325 44248fdc-6ce8-453e-bd9d-20a08f2e184c /tidb/cdc/default/__cdc_meta__/owner/22318fc84aec4352 ac0bdbd1-c7de-44d0-b033-a5927bdb4a45 /tidb/cdc/default/default/upstream/7374683499012896468 {"id":7374683499012896468,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/44248fdc-6ce8-453e-bd9d-20a08f2e184c {"id":"44248fdc-6ce8-453e-bd9d-20a08f2e184c","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/ac0bdbd1-c7de-44d0-b033-a5927bdb4a45 {"id":"ac0bdbd1-c7de-44d0-b033-a5927bdb4a45","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84aec4325 44248fdc-6ce8-453e-bd9d-20a08f2e184c /tidb/cdc/default/__cdc_meta__/owner/22318fc84aec4352 ac0bdbd1-c7de-44d0-b033-a5927bdb4a45 /tidb/cdc/default/default/upstream/7374683499012896468 {"id":7374683499012896468,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Thu May 30 14:58:47 CST 2024] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.85058507.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc3.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data3 --cluster-id default --addr 127.0.0.1:8303 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info * About to connect() to 127.0.0.1 port 8303 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8303; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 5-th time, retry later + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G13 Run cases: tiflash region_merge common_1 PROW_JOB_ID=f6375c57-288c-4d36-bd8b-6f8b308ffae9 JENKINS_NODE_COOKIE=2c665612-570b-4790-a7a3-4b83381ad059 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/ GOLANG_VERSION=1.21.10 HOSTNAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-n5fjf HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test","buildid":"1796069089735610368","prowjobid":"f6375c57-288c-4d36-bd8b-6f8b308ffae9","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#462 TEST_GROUP=G13 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089735610368 GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-jjg00 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-3335c3054d39d880f4ab7664857409e24c98a0c490a94485ce4d2e2d0717ce44 NODE_LABELS=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-jjg00 p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-n5fjf GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-n5fjf GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=462 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/tiflash/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G14 [Pipeline] sh Run cases: big_txn changefeed_finish force_replicate_table PROW_JOB_ID=f6375c57-288c-4d36-bd8b-6f8b308ffae9 JENKINS_NODE_COOKIE=acb611f5-5d51-4631-b720-dc0df45f4129 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/ GOLANG_VERSION=1.21.10 HOSTNAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-0gwt8 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test","buildid":"1796069089735610368","prowjobid":"f6375c57-288c-4d36-bd8b-6f8b308ffae9","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#462 TEST_GROUP=G14 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089735610368 GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-cmwtb GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-3335c3054d39d880f4ab7664857409e24c98a0c490a94485ce4d2e2d0717ce44 NODE_LABELS=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-cmwtb p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-0gwt8 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-0gwt8 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=462 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/big_txn/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G12 Run cases: many_pk_or_uk capture_session_done_during_task ddl_attributes PROW_JOB_ID=f6375c57-288c-4d36-bd8b-6f8b308ffae9 JENKINS_NODE_COOKIE=96131c01-462f-4445-840c-9ac954c42409 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/ GOLANG_VERSION=1.21.10 HOSTNAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8f2lb HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test","buildid":"1796069089735610368","prowjobid":"f6375c57-288c-4d36-bd8b-6f8b308ffae9","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#462 TEST_GROUP=G12 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089735610368 GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-29j3c GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-3335c3054d39d880f4ab7664857409e24c98a0c490a94485ce4d2e2d0717ce44 NODE_LABELS=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-29j3c p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8f2lb GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8f2lb GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=462 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/many_pk_or_uk/run.sh using Sink-Type: pulsar... <<================= [Pipeline] sh table partition_table.finish_mark not exists for 7-th check, retry later check diff successfully + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G11 Run cases: resolve_lock move_table autorandom generate_column PROW_JOB_ID=f6375c57-288c-4d36-bd8b-6f8b308ffae9 JENKINS_NODE_COOKIE=87dc7945-e230-41fc-98ef-78d540c98327 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/ GOLANG_VERSION=1.21.10 HOSTNAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-179cx HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test","buildid":"1796069089735610368","prowjobid":"f6375c57-288c-4d36-bd8b-6f8b308ffae9","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#462 TEST_GROUP=G11 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089735610368 GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-khh42 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-3335c3054d39d880f4ab7664857409e24c98a0c490a94485ce4d2e2d0717ce44 NODE_LABELS=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-khh42 p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-179cx GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-179cx GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=462 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/resolve_lock/run.sh using Sink-Type: pulsar... <<================= [Pipeline] // timeout + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:48 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4af45f78-30df-44a0-910b-d7ff22fdef77 {"id":"4af45f78-30df-44a0-910b-d7ff22fdef77","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84af9e5d9 4af45f78-30df-44a0-910b-d7ff22fdef77 /tidb/cdc/default/default/upstream/7374683520812005767 {"id":7374683520812005767,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4af45f78-30df-44a0-910b-d7ff22fdef77 {"id":"4af45f78-30df-44a0-910b-d7ff22fdef77","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84af9e5d9 4af45f78-30df-44a0-910b-d7ff22fdef77 /tidb/cdc/default/default/upstream/7374683520812005767 {"id":7374683520812005767,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4af45f78-30df-44a0-910b-d7ff22fdef77 {"id":"4af45f78-30df-44a0-910b-d7ff22fdef77","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84af9e5d9 4af45f78-30df-44a0-910b-d7ff22fdef77 /tidb/cdc/default/default/upstream/7374683520812005767 {"id":7374683520812005767,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5249.out cli changefeed create --start-ts=450114964225523713 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-sink-retry-test-18481?protocol=canal-json&enable-tidb-extension=true' -c changefeed-error [Pipeline] // timeout [Pipeline] } [Pipeline] } + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G17 Run cases: clustered_index processor_resolved_ts_fallback PROW_JOB_ID=f6375c57-288c-4d36-bd8b-6f8b308ffae9 JENKINS_NODE_COOKIE=39ffdc17-6406-4c17-b353-0073211302b3 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/ GOLANG_VERSION=1.21.10 HOSTNAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-vldjs HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test","buildid":"1796069089735610368","prowjobid":"f6375c57-288c-4d36-bd8b-6f8b308ffae9","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#462 TEST_GROUP=G17 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089735610368 GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-vdwm7 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-3335c3054d39d880f4ab7664857409e24c98a0c490a94485ce4d2e2d0717ce44 NODE_LABELS=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-vdwm7 p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-vldjs GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-vldjs GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=462 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/clustered_index/run.sh using Sink-Type: pulsar... <<================= [Thu May 30 14:58:49 CST 2024] <<<<<< skip test case clustered_index for pulsar! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/processor_resolved_ts_fallback/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... Create changefeed successfully! ID: changefeed-error Info: {"upstream_id":7374683520812005767,"namespace":"default","id":"changefeed-error","sink_uri":"pulsar://127.0.0.1:6650/ticdc-sink-retry-test-18481?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T14:58:49.180477352+08:00","start_ts":450114964225523713,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114964225523713,"checkpoint_ts":450114964225523713,"checkpoint_time":"2024-05-30 14:58:43.248"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... table ddl_manager.finish_mark not exists for 15-th check, retry later + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G15 Run cases: new_ci_collation batch_add_table multi_rocks PROW_JOB_ID=f6375c57-288c-4d36-bd8b-6f8b308ffae9 JENKINS_NODE_COOKIE=3b467f1d-310e-4430-a55b-ea4e58103111 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/ GOLANG_VERSION=1.21.10 HOSTNAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-s6w70 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test","buildid":"1796069089735610368","prowjobid":"f6375c57-288c-4d36-bd8b-6f8b308ffae9","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"26b901cb5b23f8c42b05e5a00514b52bef2efe24","base_link":"https://github.com/pingcap/tiflow/commit/26b901cb5b23f8c42b05e5a00514b52bef2efe24","pulls":[{"number":11200,"author":"ti-chi-bot","sha":"8f92142d7874bfad2a8db6ca100604c383fca9df","title":"*: copy seahash from blainsmith.com/go/seahash (#11193)","link":"https://github.com/pingcap/tiflow/pull/11200","commit_link":"https://github.com/pingcap/tiflow/pull/11200/commits/8f92142d7874bfad2a8db6ca100604c383fca9df","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=e64357d6ddd9be700fd5c8d87859c826b78da53a PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#462 TEST_GROUP=G15 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1796069089735610368 GOLANG_DOWNLOAD_SHA256=e330e5d977bf4f3bdc157bc46cf41afa5b13d66c914e12fd6b694ccda65fcf92 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=44e9fdd1b388d71f07c1fc94e6fccf31c512143e RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/462/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-mlzqb GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-3335c3054d39d880f4ab7664857409e24c98a0c490a94485ce4d2e2d0717ce44 NODE_LABELS=p_tiflow_release-7_5_pull_cdc_integration_pulsar_test_462-mlzqb p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-s6w70 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-s6w70 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=462 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.10.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/new_ci_collation/run.sh using Sink-Type: pulsar... <<================= check diff successfully wait process cdc.test exit for 3-th time... start tidb cluster in /tmp/tidb_cdc_test/owner_resign Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... table partition_table.finish_mark exists check diff successfully + set +x [Thu May 30 14:58:50 CST 2024] <<<<<< START Pulsar consumer in changefeed_error case >>>>>> check_changefeed_state http://127.0.0.1:2379 changefeed-error failed [CDC:ErrStartTsBeforeGC] + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error + expected_state=failed + error_msg='[CDC:ErrStartTsBeforeGC]' + tls_dir='[CDC:ErrStartTsBeforeGC]' + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error -s + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info * About to connect() to 127.0.0.1 port 8303 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8303 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8303 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:58:50 GMT < Content-Length: 1144 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/329da4fe-eb61-431f-950a-dec5dd2a0164 {"id":"329da4fe-eb61-431f-950a-dec5dd2a0164","address":"127.0.0.1:8303","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/44248fdc-6ce8-453e-bd9d-20a08f2e184c {"id":"44248fdc-6ce8-453e-bd9d-20a08f2e184c","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/ac0bdbd1-c7de-44d0-b033-a5927bdb4a45 {"id":"ac0bdbd1-c7de-44d0-b033-a5927bdb4a45","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84aec4325 44248fdc-6ce8-453e-bd9d-20a08f2e184c /tidb/cdc/default/__cdc_meta__/owner/22318fc84aec4352 ac0bdbd1-c7de-44d0-b033-a5927bdb4a45 /tidb/cdc/default/__cdc_meta__/owner/22318fc84aec4368 329da4fe-eb61-431f-950a-dec5dd2a0164 /tidb/cdc/default/default/upstream/7374683499012896468 {"id":7374683499012896468,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/329da4fe-eb61-431f-950a-dec5dd2a0164 {"id":"329da4fe-eb61-431f-950a-dec5dd2a0164","address":"127.0.0.1:8303","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/44248fdc-6ce8-453e-bd9d-20a08f2e184c {"id":"44248fdc-6ce8-453e-bd9d-20a08f2e184c","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/ac0bdbd1-c7de-44d0-b033-a5927bdb4a45 {"id":"ac0bdbd1-c7de-44d0-b033-a5927bdb4a45","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84aec4325 44248fdc-6ce8-453e-bd9d-20a08f2e184c /tidb/cdc/default/__cdc_meta__/owner/22318fc84aec4352 ac0bdbd1-c7de-44d0-b033-a5927bdb4a45 /tidb/cdc/default/__cdc_meta__/owner/22318fc84aec4368 329da4fe-eb61-431f-950a-dec5dd2a0164 /tidb/cdc/default/default/upstream/7374683499012896468 {"id":7374683499012896468,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/329da4fe-eb61-431f-950a-dec5dd2a0164 {"id":"329da4fe-eb61-431f-950a-dec5dd2a0164","address":"127.0.0.1:8303","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/44248fdc-6ce8-453e-bd9d-20a08f2e184c {"id":"44248fdc-6ce8-453e-bd9d-20a08f2e184c","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/ac0bdbd1-c7de-44d0-b033-a5927bdb4a45 {"id":"ac0bdbd1-c7de-44d0-b033-a5927bdb4a45","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84aec4325 44248fdc-6ce8-453e-bd9d-20a08f2e184c /tidb/cdc/default/__cdc_meta__/owner/22318fc84aec4352 ac0bdbd1-c7de-44d0-b033-a5927bdb4a45 /tidb/cdc/default/__cdc_meta__/owner/22318fc84aec4368 329da4fe-eb61-431f-950a-dec5dd2a0164 /tidb/cdc/default/default/upstream/7374683499012896468 {"id":7374683499012896468,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.cli.8550.out cli changefeed create --start-ts=450114963034865665 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-multi-capture-test-4753?protocol=canal-json&enable-tidb-extension=true' --server=127.0.0.1:8301 Create changefeed successfully! ID: 7bee0fa5-774a-421c-aaf2-abdaae302a6f Info: {"upstream_id":7374683499012896468,"namespace":"default","id":"7bee0fa5-774a-421c-aaf2-abdaae302a6f","sink_uri":"pulsar://127.0.0.1:6650/ticdc-multi-capture-test-4753?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T14:58:50.588935597+08:00","start_ts":450114963034865665,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114963034865665,"checkpoint_ts":450114963034865665,"checkpoint_time":"2024-05-30 14:58:38.706"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 14:58:50 CST 2024] <<<<<< run test case split_region success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/tiflash Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... + info='{ "upstream_id": 7374683520812005767, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 450114964225523713, "checkpoint_time": "2024-05-30 14:58:43.248", "error": { "time": "2024-05-30T14:58:49.269046658+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 450114964225523413 is earlier than or equal to GC safepoint at 450114964225523713" } }' + echo '{ "upstream_id": 7374683520812005767, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 450114964225523713, "checkpoint_time": "2024-05-30 14:58:43.248", "error": { "time": "2024-05-30T14:58:49.269046658+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 450114964225523413 is earlier than or equal to GC safepoint at 450114964225523713" } }' { "upstream_id": 7374683520812005767, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 450114964225523713, "checkpoint_time": "2024-05-30 14:58:43.248", "error": { "time": "2024-05-30T14:58:49.269046658+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 450114964225523413 is earlier than or equal to GC safepoint at 450114964225523713" } } ++ echo '{' '"upstream_id":' 7374683520812005767, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"failed",' '"checkpoint_tso":' 450114964225523713, '"checkpoint_time":' '"2024-05-30' '14:58:43.248",' '"error":' '{' '"time":' '"2024-05-30T14:58:49.269046658+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]fail' to create or maintain changefeed because start-ts 450114964225523413 is earlier than or equal to GC safepoint at '450114964225523713"' '}' '}' ++ jq -r .state + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ jq -r .error.message ++ echo '{' '"upstream_id":' 7374683520812005767, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"failed",' '"checkpoint_tso":' 450114964225523713, '"checkpoint_time":' '"2024-05-30' '14:58:43.248",' '"error":' '{' '"time":' '"2024-05-30T14:58:49.269046658+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]fail' to create or maintain changefeed because start-ts 450114964225523413 is earlier than or equal to GC safepoint at '450114964225523713"' '}' '}' + message='[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 450114964225523413 is earlier than or equal to GC safepoint at 450114964225523713' + [[ ! [CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 450114964225523413 is earlier than or equal to GC safepoint at 450114964225523713 =~ \[CDC:ErrStartTsBeforeGC] ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5344.out cli changefeed resume -c changefeed-error wait process cdc.test exit for 1-th time... check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 16-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/big_txn Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... wait process cdc.test exit for 2-th time... find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/capture_session_done_during_task/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + set +x [Thu May 30 14:58:52 CST 2024] <<<<<< START Pulsar consumer in multi_capture case >>>>>> table multi_capture_1.usertable not exists for 1-th check, retry later The 1 times to try to start tidb cluster... wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 14:58:52 CST 2024] <<<<<< run test case partition_table success! >>>>>> Verifying downstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/processor_resolved_ts_fallback Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 17-th check, retry later + set +x table changefeed_error.usertable not exists for 1-th check, retry later check diff failed 2-th time, retry later table multi_capture_1.usertable exists table multi_capture_2.usertable not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/batch_add_table/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table ddl_manager.finish_mark not exists for 18-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/resolve_lock Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... table changefeed_error.usertable exists check diff failed 1-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release check diff failed 3-th time, retry later [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc already exists) table multi_capture_2.usertable exists table multi_capture_3.usertable not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212d77940019 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sv1w0, pid:6256, start at 2024-05-30 14:58:53.587974801 +0800 CST m=+5.196327267 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:53.596 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:53.591 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:53.591 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212d77940019 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sv1w0, pid:6256, start at 2024-05-30 14:58:53.587974801 +0800 CST m=+5.196327267 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:53.596 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:53.591 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:53.591 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212d7a740014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sv1w0, pid:6331, start at 2024-05-30 14:58:53.755515589 +0800 CST m=+5.314860490 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:00:53.762 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:58:53.725 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:48:53.725 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Logging trace to /tmp/tidb_cdc_test/event_filter/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/event_filter/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table ddl_manager.finish_mark not exists for 19-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully ***************** properties ***************** "insertproportion"="0" "mysql.db"="changefeed_error" "operationcount"="0" "workload"="core" "updateproportion"="0" "requestdistribution"="uniform" "recordcount"="20" "readproportion"="0" "readallfields"="true" "dotransactions"="false" "mysql.host"="127.0.0.1" "threadcount"="4" "mysql.user"="root" "scanproportion"="0" "mysql.port"="4000" ********************************************** Run finished, takes 4.330882ms INSERT - Takes(s): 0.0, Count: 20, OPS: 7324.2, Avg(us): 767, Min(us): 399, Max(us): 2028, 95th(us): 2000, 99th(us): 3000 check diff successfully {"id":"4af45f78-30df-44a0-910b-d7ff22fdef77","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/4af45f78-30df-44a0-910b-d7ff22fdef77 + [[ /tidb/cdc/default/__cdc_meta__/capture/4af45f78-30df-44a0-910b-d7ff22fdef77 =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/4af45f78-30df-44a0-910b-d7ff22fdef77' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/4af45f78-30df-44a0-910b-d7ff22fdef77 + echo 'check failed' check failed + exit 1 run task failed 1-th time, retry later table multi_capture_3.usertable exists table multi_capture_4.usertable not exists for 1-th check, retry later check diff failed 4-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Thu May 30 14:58:58 CST 2024] <<<<<< START cdc server in event_filter case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.event_filter.75707572.out server --log-file /tmp/tidb_cdc_test/event_filter/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/event_filter/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release start tidb cluster in /tmp/tidb_cdc_test/capture_session_done_during_task Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/4af45f78-30df-44a0-910b-d7ff22fdef77 + [[ /tidb/cdc/default/__cdc_meta__/capture/4af45f78-30df-44a0-910b-d7ff22fdef77 =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/4af45f78-30df-44a0-910b-d7ff22fdef77' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/4af45f78-30df-44a0-910b-d7ff22fdef77 + echo 'check failed' check failed + exit 1 run task failed 2-th time, retry later table multi_capture_4.usertable not exists for 2-th check, retry later check diff failed 5-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 20-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/batch_add_table Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:01 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ea32b859-ac7c-4b07-a3a1-100a7147b5c1 {"id":"ea32b859-ac7c-4b07-a3a1-100a7147b5c1","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b35d4bf ea32b859-ac7c-4b07-a3a1-100a7147b5c1 /tidb/cdc/default/default/upstream/7374683585581905336 {"id":7374683585581905336,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ea32b859-ac7c-4b07-a3a1-100a7147b5c1 {"id":"ea32b859-ac7c-4b07-a3a1-100a7147b5c1","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b35d4bf ea32b859-ac7c-4b07-a3a1-100a7147b5c1 /tidb/cdc/default/default/upstream/7374683585581905336 {"id":7374683585581905336,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ea32b859-ac7c-4b07-a3a1-100a7147b5c1 {"id":"ea32b859-ac7c-4b07-a3a1-100a7147b5c1","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b35d4bf ea32b859-ac7c-4b07-a3a1-100a7147b5c1 /tidb/cdc/default/default/upstream/7374683585581905336 {"id":7374683585581905336,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.event_filter.cli.7615.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-event-filter-15099?protocol=canal-json&enable-tidb-extension=true' --server=127.0.0.1:8300 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/event_filter/conf/cf.toml ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table multi_capture_4.usertable exists check diff failed 6-th time, retry later \033[0;36m<<< Run all test success >>>\033[0m table ddl_manager.finish_mark not exists for 21-th check, retry later Create changefeed successfully! ID: b020b4a4-423c-4f01-af08-62e7082c2d3a Info: {"upstream_id":7374683585581905336,"namespace":"default","id":"b020b4a4-423c-4f01-af08-62e7082c2d3a","sink_uri":"pulsar://127.0.0.1:6650/ticdc-event-filter-15099?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T14:59:02.286552151+08:00","start_ts":450114969178210307,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["event_filter.*"],"event_filters":[{"matcher":["event_filter.t1"],"ignore_event":["drop table","delete"],"ignore_sql":null,"ignore_insert_value_expr":"id = 2 or city = 'tokyo'","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""},{"matcher":["event_filter.t_truncate"],"ignore_event":["truncate table"],"ignore_sql":null,"ignore_insert_value_expr":"","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""},{"matcher":["event_filter.t_alter"],"ignore_event":["alter table"],"ignore_sql":null,"ignore_insert_value_expr":"","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""}]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114969178210307,"checkpoint_ts":450114969178210307,"checkpoint_time":"2024-05-30 14:59:02.141"} PASS check diff failed 1-th time, retry later coverage: 2.5% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x [Thu May 30 14:59:03 CST 2024] <<<<<< START Pulsar consumer in event_filter case >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/4af45f78-30df-44a0-910b-d7ff22fdef77 + [[ /tidb/cdc/default/__cdc_meta__/capture/4af45f78-30df-44a0-910b-d7ff22fdef77 =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/4af45f78-30df-44a0-910b-d7ff22fdef77' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/4af45f78-30df-44a0-910b-d7ff22fdef77 + echo 'check failed' check failed + exit 1 run task failed 3-th time, retry later table ddl_manager.finish_mark not exists for 22-th check, retry later check diff successfully ***************** properties ***************** "mysql.db"="multi_capture_1" "requestdistribution"="uniform" "readproportion"="0" "mysql.host"="127.0.0.1" "updateproportion"="0" "recordcount"="20" "readallfields"="true" "mysql.user"="root" "threadcount"="2" "workload"="core" "dotransactions"="false" "mysql.port"="4000" "scanproportion"="0" "insertproportion"="0" "operationcount"="0" ********************************************** Run finished, takes 16.684144ms INSERT - Takes(s): 0.0, Count: 20, OPS: 1320.2, Avg(us): 1607, Min(us): 486, Max(us): 5689, 95th(us): 6000, 99th(us): 6000 ***************** properties ***************** "mysql.port"="4000" "requestdistribution"="uniform" "scanproportion"="0" "threadcount"="2" "dotransactions"="false" "readallfields"="true" "recordcount"="20" "operationcount"="0" "mysql.user"="root" "insertproportion"="0" "readproportion"="0" "workload"="core" "updateproportion"="0" "mysql.host"="127.0.0.1" "mysql.db"="multi_capture_2" ********************************************** Run finished, takes 11.563309ms INSERT - Takes(s): 0.0, Count: 20, OPS: 1936.7, Avg(us): 988, Min(us): 531, Max(us): 1691, 95th(us): 2000, 99th(us): 2000 ***************** properties ***************** "requestdistribution"="uniform" "dotransactions"="false" "mysql.db"="multi_capture_3" "scanproportion"="0" "readproportion"="0" "insertproportion"="0" "operationcount"="0" "mysql.port"="4000" "updateproportion"="0" "threadcount"="2" "workload"="core" "recordcount"="20" "readallfields"="true" "mysql.host"="127.0.0.1" "mysql.user"="root" ********************************************** Run finished, takes 16.772324ms INSERT - Takes(s): 0.0, Count: 20, OPS: 1300.7, Avg(us): 1491, Min(us): 571, Max(us): 5548, 95th(us): 5000, 99th(us): 6000 ***************** properties ***************** "recordcount"="20" "mysql.port"="4000" "dotransactions"="false" "updateproportion"="0" "readproportion"="0" "mysql.host"="127.0.0.1" "mysql.db"="multi_capture_4" "operationcount"="0" "workload"="core" "threadcount"="2" "mysql.user"="root" "readallfields"="true" "insertproportion"="0" "requestdistribution"="uniform" "scanproportion"="0" ********************************************** Run finished, takes 14.15718ms INSERT - Takes(s): 0.0, Count: 20, OPS: 1567.6, Avg(us): 1217, Min(us): 580, Max(us): 2313, 95th(us): 2000, 99th(us): 3000 check diff successfully VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212dfbe40016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-k5r94, pid:1185, start at 2024-05-30 14:59:02.050628364 +0800 CST m=+5.716751288 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:02.060 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:02.059 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:02.059 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212dfbe40016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-k5r94, pid:1185, start at 2024-05-30 14:59:02.050628364 +0800 CST m=+5.716751288 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:02.060 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:02.059 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:02.059 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212dff500002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-k5r94, pid:1262, start at 2024-05-30 14:59:02.229575865 +0800 CST m=+5.841031298 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:02.240 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:02.228 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:02.228 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/owner_resign/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/owner_resign/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 1-th time, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212e00f40017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-n5fjf, pid:1209, start at 2024-05-30 14:59:02.366746763 +0800 CST m=+5.444033033 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:02.374 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:02.333 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:02.333 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212e00f40017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-n5fjf, pid:1209, start at 2024-05-30 14:59:02.366746763 +0800 CST m=+5.444033033 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:02.374 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:02.333 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:02.333 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212e0280000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-n5fjf, pid:1285, start at 2024-05-30 14:59:02.445404169 +0800 CST m=+5.458015089 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:02.453 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:02.432 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:02.432 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/tiflash/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/tiflash/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212e1cd80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-vldjs, pid:1159, start at 2024-05-30 14:59:04.14680842 +0800 CST m=+6.839606279 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:04.154 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:04.118 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:04.118 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212e1cd80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-vldjs, pid:1159, start at 2024-05-30 14:59:04.14680842 +0800 CST m=+6.839606279 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:04.154 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:04.118 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:04.118 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212e1b400014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-vldjs, pid:1242, start at 2024-05-30 14:59:04.064029594 +0800 CST m=+6.684723566 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:04.073 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:04.066 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:04.066 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212e09f80014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-0gwt8, pid:1196, start at 2024-05-30 14:59:02.957906403 +0800 CST m=+5.500193794 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:02.967 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:02.960 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:02.960 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212e09f80014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-0gwt8, pid:1196, start at 2024-05-30 14:59:02.957906403 +0800 CST m=+5.500193794 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:02.967 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:02.960 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:02.960 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212e0b7c0015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-0gwt8, pid:1274, start at 2024-05-30 14:59:03.051523348 +0800 CST m=+5.516623015 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:03.061 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:03.057 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:03.057 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/big_txn/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/big_txn/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/big_txn/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/big_txn/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/big_txn/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 23-th check, retry later [Pipeline] // stage [Pipeline] // stage [Thu May 30 14:59:06 CST 2024] <<<<<< START cdc server in owner_resign case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.25862588.out server --log-file /tmp/tidb_cdc_test/owner_resign/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/owner_resign/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff successfully wait process cdc.test exit for 1-th time... table event_filter.t1 does not exists table event_filter.t1 exists table event_filter.t_normal not exists for 1-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tiflash.cli.2595.out cli tso query --pd=http://127.0.0.1:2379 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] } [Pipeline] } [Thu May 30 14:59:07 CST 2024] <<<<<< START cdc server in processor_resolved_ts_fallback case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkExecDMLError=return(true);github.com/pingcap/tiflow/cdc/sink/dmlsink/mq/dmlproducer/KafkaSinkAsyncSendError=return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.25932595.out server --log-file /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... table ddl_manager.finish_mark not exists for 24-th check, retry later [Pipeline] // cache ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 3-th time... [Thu May 30 14:59:07 CST 2024] <<<<<< START cdc server in big_txn case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.big_txn.25662568.out server --log-file /tmp/tidb_cdc_test/big_txn/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/big_txn/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 2-th time, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 14:59:08 CST 2024] <<<<<< run test case multi_capture success! >>>>>> table event_filter.t_normal exists table event_filter.t_truncate not exists for 1-th check, retry later + set +x + tso='450114970618167297 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114970618167297 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 14:59:09 CST 2024] <<<<<< START cdc server in tiflash case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tiflash.26232625.out server --log-file /tmp/tidb_cdc_test/tiflash/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/tiflash/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212e468c0017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-179cx, pid:1318, start at 2024-05-30 14:59:06.823338556 +0800 CST m=+5.528260677 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:06.831 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:06.836 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:06.836 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212e468c0017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-179cx, pid:1318, start at 2024-05-30 14:59:06.823338556 +0800 CST m=+5.528260677 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:06.831 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:06.836 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:06.836 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212e46900015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-179cx, pid:1404, start at 2024-05-30 14:59:06.816548007 +0800 CST m=+5.434417333 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:06.824 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:06.788 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:06.788 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/resolve_lock/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/resolve_lock/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:09 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b3bf1dab-31b1-436b-af8a-0040bb261270 {"id":"b3bf1dab-31b1-436b-af8a-0040bb261270","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b52e1bd b3bf1dab-31b1-436b-af8a-0040bb261270 /tidb/cdc/default/default/upstream/7374683618857110233 {"id":7374683618857110233,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b3bf1dab-31b1-436b-af8a-0040bb261270 {"id":"b3bf1dab-31b1-436b-af8a-0040bb261270","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b52e1bd b3bf1dab-31b1-436b-af8a-0040bb261270 /tidb/cdc/default/default/upstream/7374683618857110233 {"id":7374683618857110233,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b3bf1dab-31b1-436b-af8a-0040bb261270 {"id":"b3bf1dab-31b1-436b-af8a-0040bb261270","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b52e1bd b3bf1dab-31b1-436b-af8a-0040bb261270 /tidb/cdc/default/default/upstream/7374683618857110233 {"id":7374683618857110233,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc cli capture list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' 1 run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.cli.2670.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-owner-resign-test-21608?protocol=canal-json&enable-tidb-extension=true' --server=127.0.0.1:8301 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] } [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc already exists) [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc already exists) check diff failed 3-th time, retry later Create changefeed successfully! ID: 1f31080e-dea6-452a-928f-f2520071d53b Info: {"upstream_id":7374683618857110233,"namespace":"default","id":"1f31080e-dea6-452a-928f-f2520071d53b","sink_uri":"pulsar://127.0.0.1:6650/ticdc-owner-resign-test-21608?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T14:59:10.376799563+08:00","start_ts":450114971293450241,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114971293450241,"checkpoint_ts":450114971293450241,"checkpoint_time":"2024-05-30 14:59:10.210"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 25-th check, retry later [Pipeline] // container [Pipeline] // container [Pipeline] } [Pipeline] } check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info= + [[ '' =~ capture ]] + echo 'check pass' check pass + exit 0 run task successfully check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Thu May 30 14:59:09 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/NewChangefeedRetryError=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.55735575.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:10 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1117e6c3-c62a-492a-81f8-fd588c23798e {"id":"1117e6c3-c62a-492a-81f8-fd588c23798e","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b5559ba 1117e6c3-c62a-492a-81f8-fd588c23798e /tidb/cdc/default/default/upstream/7374683620176761798 {"id":7374683620176761798,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1117e6c3-c62a-492a-81f8-fd588c23798e {"id":"1117e6c3-c62a-492a-81f8-fd588c23798e","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b5559ba 1117e6c3-c62a-492a-81f8-fd588c23798e /tidb/cdc/default/default/upstream/7374683620176761798 {"id":7374683620176761798,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1117e6c3-c62a-492a-81f8-fd588c23798e {"id":"1117e6c3-c62a-492a-81f8-fd588c23798e","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b5559ba 1117e6c3-c62a-492a-81f8-fd588c23798e /tidb/cdc/default/default/upstream/7374683620176761798 {"id":7374683620176761798,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Thu May 30 14:59:10 CST 2024] <<<<<< START cdc server in big_txn case >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:10 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c68f1aad-b5b8-4d25-b1a0-511900d33167 {"id":"c68f1aad-b5b8-4d25-b1a0-511900d33167","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b58b4bb c68f1aad-b5b8-4d25-b1a0-511900d33167 /tidb/cdc/default/default/upstream/7374683623901191057 {"id":7374683623901191057,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c68f1aad-b5b8-4d25-b1a0-511900d33167 {"id":"c68f1aad-b5b8-4d25-b1a0-511900d33167","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b58b4bb c68f1aad-b5b8-4d25-b1a0-511900d33167 /tidb/cdc/default/default/upstream/7374683623901191057 {"id":7374683623901191057,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c68f1aad-b5b8-4d25-b1a0-511900d33167 {"id":"c68f1aad-b5b8-4d25-b1a0-511900d33167","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b58b4bb c68f1aad-b5b8-4d25-b1a0-511900d33167 /tidb/cdc/default/default/upstream/7374683623901191057 {"id":7374683623901191057,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.cli.2635.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-processor-resolved-ts-fallback-test-15758?protocol=canal-json&enable-tidb-extension=true' --server=127.0.0.1:8301 table event_filter.t_truncate exists table event_filter.t_alter not exists for 1-th check, retry later + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.big_txn.26102612.out server --log-file /tmp/tidb_cdc_test/big_txn/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/big_txn/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:11 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1117e6c3-c62a-492a-81f8-fd588c23798e {"id":"1117e6c3-c62a-492a-81f8-fd588c23798e","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b5559ba 1117e6c3-c62a-492a-81f8-fd588c23798e /tidb/cdc/default/default/upstream/7374683620176761798 {"id":7374683620176761798,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1117e6c3-c62a-492a-81f8-fd588c23798e {"id":"1117e6c3-c62a-492a-81f8-fd588c23798e","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b5559ba 1117e6c3-c62a-492a-81f8-fd588c23798e /tidb/cdc/default/default/upstream/7374683620176761798 {"id":7374683620176761798,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1117e6c3-c62a-492a-81f8-fd588c23798e {"id":"1117e6c3-c62a-492a-81f8-fd588c23798e","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b5559ba 1117e6c3-c62a-492a-81f8-fd588c23798e /tidb/cdc/default/default/upstream/7374683620176761798 {"id":7374683620176761798,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.big_txn.cli.2639.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-big-txn-test-3487?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: 57cbb323-f9fa-474b-be02-12d72c8e6d75 Info: {"upstream_id":7374683623901191057,"namespace":"default","id":"57cbb323-f9fa-474b-be02-12d72c8e6d75","sink_uri":"pulsar://127.0.0.1:6650/ticdc-processor-resolved-ts-fallback-test-15758?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T14:59:11.269795445+08:00","start_ts":450114971518107656,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114971518107656,"checkpoint_ts":450114971518107656,"checkpoint_time":"2024-05-30 14:59:11.067"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [Pipeline] // dir [Pipeline] // cache [Pipeline] // cache [Pipeline] } [Pipeline] } [Pipeline] } Create changefeed successfully! ID: 7040b2d7-edcf-4b3c-b7b0-88c3136b3ae6 Info: {"upstream_id":7374683620176761798,"namespace":"default","id":"7040b2d7-edcf-4b3c-b7b0-88c3136b3ae6","sink_uri":"pulsar://127.0.0.1:6650/ticdc-big-txn-test-3487?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T14:59:11.616933562+08:00","start_ts":450114971608023044,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114971608023044,"checkpoint_ts":450114971608023044,"checkpoint_time":"2024-05-30 14:59:11.410"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [Pipeline] // withEnv [Pipeline] // withEnv + set +x [Thu May 30 14:59:11 CST 2024] <<<<<< START Pulsar consumer in owner_resign case >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.cli.2640.out cli tso query --pd=http://127.0.0.1:2379 check diff failed 4-th time, retry later table ddl_manager.finish_mark not exists for 26-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:12 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8d1f9612-71e7-427c-b6de-dd2d0968301f {"id":"8d1f9612-71e7-427c-b6de-dd2d0968301f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b5319c8 8d1f9612-71e7-427c-b6de-dd2d0968301f /tidb/cdc/default/default/upstream/7374683608275589453 {"id":7374683608275589453,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8d1f9612-71e7-427c-b6de-dd2d0968301f {"id":"8d1f9612-71e7-427c-b6de-dd2d0968301f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b5319c8 8d1f9612-71e7-427c-b6de-dd2d0968301f /tidb/cdc/default/default/upstream/7374683608275589453 {"id":7374683608275589453,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8d1f9612-71e7-427c-b6de-dd2d0968301f {"id":"8d1f9612-71e7-427c-b6de-dd2d0968301f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b5319c8 8d1f9612-71e7-427c-b6de-dd2d0968301f /tidb/cdc/default/default/upstream/7374683608275589453 {"id":7374683608275589453,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tiflash.cli.2670.out cli changefeed create --start-ts=450114970618167297 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-tiflash-test-9132?protocol=canal-json&enable-tidb-extension=true' + set +x [Thu May 30 14:59:12 CST 2024] <<<<<< START Pulsar consumer in processor_resolved_ts_fallback case >>>>>> Create changefeed successfully! ID: ed7d8d8d-9a59-4817-a4c6-80f1c8beb652 Info: {"upstream_id":7374683608275589453,"namespace":"default","id":"ed7d8d8d-9a59-4817-a4c6-80f1c8beb652","sink_uri":"pulsar://127.0.0.1:6650/ticdc-tiflash-test-9132?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T14:59:12.911055868+08:00","start_ts":450114970618167297,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114970618167297,"checkpoint_ts":450114970618167297,"checkpoint_time":"2024-05-30 14:59:07.634"} PASS cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' 1 run task successfully table processor_resolved_ts_fallback.t1 not exists for 1-th check, retry later table event_filter.t_alter exists [Pipeline] } coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x [Pipeline] } cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' 1 run task successfully table owner_resign.t1 exists check diff failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:13 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d77a75e5-257d-431b-8fe8-e7ebe54555f3 {"id":"d77a75e5-257d-431b-8fe8-e7ebe54555f3","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318fc84af9e6a2 d77a75e5-257d-431b-8fe8-e7ebe54555f3 /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7374683520812005767,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"pulsar://127.0.0.1:6650/ticdc-sink-retry-test-18481?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T14:58:49.180477352+08:00","start-ts":450114964225523713,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"pending","error":{"time":"2024-05-30T14:59:10.74341769+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114971421638662} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":450114967869587466,"min-table-barrier-ts":450114967869587466,"admin-job-type":1} /tidb/cdc/default/default/upstream/7374683520812005767 {"id":7374683520812005767,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d77a75e5-257d-431b-8fe8-e7ebe54555f3 {"id":"d77a75e5-257d-431b-8fe8-e7ebe54555f3","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318fc84af9e6a2 d77a75e5-257d-431b-8fe8-e7ebe54555f3 /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7374683520812005767,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"pulsar://127.0.0.1:6650/ticdc-sink-retry-test-18481?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T14:58:49.180477352+08:00","start-ts":450114964225523713,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"pending","error":{"time":"2024-05-30T14:59:10.74341769+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114971421638662} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":450114967869587466,"min-table-barrier-ts":450114967869587466,"admin-job-type":1} /tidb/cdc/default/default/upstream/7374683520812005767 {"id":7374683520812005767,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d77a75e5-257d-431b-8fe8-e7ebe54555f3 {"id":"d77a75e5-257d-431b-8fe8-e7ebe54555f3","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318fc84af9e6a2 d77a75e5-257d-431b-8fe8-e7ebe54555f3 /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7374683520812005767,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"pulsar://127.0.0.1:6650/ticdc-sink-retry-test-18481?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T14:58:49.180477352+08:00","start-ts":450114964225523713,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"pending","error":{"time":"2024-05-30T14:59:10.74341769+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114971421638662} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":450114967869587466,"min-table-barrier-ts":450114967869587466,"admin-job-type":1} /tidb/cdc/default/default/upstream/7374683520812005767 {"id":7374683520812005767,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-error warning failpoint injected retriable error + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error + expected_state=warning + error_msg=failpoint + tls_dir=error + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error -s + info='{ "upstream_id": 7374683520812005767, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 450114967869587466, "checkpoint_time": "2024-05-30 14:58:57.149", "error": { "time": "2024-05-30T14:59:10.74341769+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } }' + echo '{ "upstream_id": 7374683520812005767, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 450114967869587466, "checkpoint_time": "2024-05-30 14:58:57.149", "error": { "time": "2024-05-30T14:59:10.74341769+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } }' { "upstream_id": 7374683520812005767, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 450114967869587466, "checkpoint_time": "2024-05-30 14:58:57.149", "error": { "time": "2024-05-30T14:59:10.74341769+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } } ++ echo '{' '"upstream_id":' 7374683520812005767, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"warning",' '"checkpoint_tso":' 450114967869587466, '"checkpoint_time":' '"2024-05-30' '14:58:57.149",' '"error":' '{' '"time":' '"2024-05-30T14:59:10.74341769+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrOwnerUnknown",' '"message":' '"failpoint' injected retriable 'error"' '}' '}' ++ jq -r .state + state=warning + [[ ! warning == \w\a\r\n\i\n\g ]] ++ echo '{' '"upstream_id":' 7374683520812005767, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"warning",' '"checkpoint_tso":' 450114967869587466, '"checkpoint_time":' '"2024-05-30' '14:58:57.149",' '"error":' '{' '"time":' '"2024-05-30T14:59:10.74341769+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrOwnerUnknown",' '"message":' '"failpoint' injected retriable 'error"' '}' '}' ++ jq -r .error.message + message='failpoint injected retriable error' + [[ ! failpoint injected retriable error =~ failpoint ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5654.out cli changefeed remove -c changefeed-error + set +x + tso='450114971772387332 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114971772387332 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 14:59:13 CST 2024] <<<<<< START cdc server in resolve_lock case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + GO_FAILPOINTS= + etcd_info_msg='etcd info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.26732675.out server --log-file /tmp/tidb_cdc_test/resolve_lock/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/resolve_lock/cdc_data --cluster-id default + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // timeout [Pipeline] // dir [Pipeline] // dir [Pipeline] } [Pipeline] } [Pipeline] } Changefeed remove successfully. ID: changefeed-error CheckpointTs: 450114967869587466 SinkURI: pulsar://127.0.0.1:6650/ticdc-sink-retry-test-18481?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 27-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212e965c0007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-s6w70, pid:1326, start at 2024-05-30 14:59:11.899877875 +0800 CST m=+7.103807441 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:11.909 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:11.895 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:11.895 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212e965c0007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-s6w70, pid:1326, start at 2024-05-30 14:59:11.899877875 +0800 CST m=+7.103807441 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:11.909 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:11.895 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:11.895 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f212e97340015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-s6w70, pid:1391, start at 2024-05-30 14:59:11.994146946 +0800 CST m=+7.139776842 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:12.001 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:12.000 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:12.000 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ***************** properties ***************** "mysql.db"="big_txn" "mysql.host"="127.0.0.1" "mysql.user"="root" "operationcount"="0" "insertproportion"="0" "recordcount"="5000" "threadcount"="1" "requestdistribution"="uniform" "dotransactions"="false" "readproportion"="0" "scanproportion"="0" "readallfields"="true" "updateproportion"="0" "mysql.port"="4000" "workload"="core" "fieldcount"="100" ********************************************** [2024/05/30 14:59:14.618 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001954b40/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] check diff successfully table event_filter.finish_mark exists check diff failed 1-th time, retry later [Pipeline] // node [Pipeline] // node [Pipeline] } [Pipeline] } table processor_resolved_ts_fallback.t1 exists [Thu May 30 14:59:15 CST 2024] <<<<<< START cdc server in processor_resolved_ts_fallback case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.27702772.out server --log-file /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] // stage [Pipeline] // timeout + set +x [Thu May 30 14:59:15 CST 2024] <<<<<< START Pulsar consumer in tiflash case >>>>>> [Pipeline] // timeout ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later + set +x check_no_changefeed 127.0.0.1:2379 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] // podTemplate [Pipeline] // podTemplate [Pipeline] } [Pipeline] } check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 28-th check, retry later wait process cdc.test exit for 1-th time... [Pipeline] // container [Pipeline] // stage [Pipeline] // stage [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] // withEnv [Pipeline] // withEnv [Pipeline] } [Pipeline] } [Pipeline] // withEnv [Pipeline] // container + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:16 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b8cce7bc-c160-45be-92c0-e01a66f97d7f {"id":"b8cce7bc-c160-45be-92c0-e01a66f97d7f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b644dc2 b8cce7bc-c160-45be-92c0-e01a66f97d7f /tidb/cdc/default/default/upstream/7374683640738619107 {"id":7374683640738619107,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b8cce7bc-c160-45be-92c0-e01a66f97d7f {"id":"b8cce7bc-c160-45be-92c0-e01a66f97d7f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b644dc2 b8cce7bc-c160-45be-92c0-e01a66f97d7f /tidb/cdc/default/default/upstream/7374683640738619107 {"id":7374683640738619107,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b8cce7bc-c160-45be-92c0-e01a66f97d7f {"id":"b8cce7bc-c160-45be-92c0-e01a66f97d7f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b644dc2 b8cce7bc-c160-45be-92c0-e01a66f97d7f /tidb/cdc/default/default/upstream/7374683640738619107 {"id":7374683640738619107,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.cli.2723.out cli changefeed create --start-ts=450114971772387332 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-resolve-lock-test-16371?protocol=canal-json&enable-tidb-extension=true' wait process cdc.test exit for 2-th time... [Pipeline] // container [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] // stage Create changefeed successfully! ID: 12f7a349-9e80-46a3-b7e0-67a3e7df9a24 Info: {"upstream_id":7374683640738619107,"namespace":"default","id":"12f7a349-9e80-46a3-b7e0-67a3e7df9a24","sink_uri":"pulsar://127.0.0.1:6650/ticdc-resolve-lock-test-16371?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T14:59:17.126602404+08:00","start_ts":450114971772387332,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114971772387332,"checkpoint_ts":450114971772387332,"checkpoint_time":"2024-05-30 14:59:12.037"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [Pipeline] // stage [Pipeline] } [Pipeline] } [Pipeline] // node [Pipeline] // withEnv [Pipeline] // withEnv [Pipeline] } check diff failed 2-th time, retry later [Pipeline] } wait process cdc.test exit for 3-th time... [Pipeline] } check diff successfully [Thu May 30 14:59:17 CST 2024] <<<<<< START cdc server in owner_resign case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerDontUpdateCheckpoint=return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.29172919.out server --log-file /tmp/tidb_cdc_test/owner_resign/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/owner_resign/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] // podTemplate [Pipeline] // node [Pipeline] // node [Pipeline] } [Pipeline] } [Pipeline] } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:18 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/57cbb323-f9fa-474b-be02-12d72c8e6d75 {UpstreamID:7374683623901191057 Namespace:default ID:57cbb323-f9fa-474b-be02-12d72c8e6d75 SinkURI:pulsar://127.0.0.1:6650/ticdc-processor-resolved-ts-fallback-test-15758?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 14:59:11.269795445 +0800 CST StartTs:450114971518107656 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0038d01b0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114971570798593} {CheckpointTs:450114973182984195 MinTableBarrierTs:450114973182984195 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/49de05a5-3f1e-44bf-9847-3b5971b2b008 {"id":"49de05a5-3f1e-44bf-9847-3b5971b2b008","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/c68f1aad-b5b8-4d25-b1a0-511900d33167 {"id":"c68f1aad-b5b8-4d25-b1a0-511900d33167","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b58b4bb c68f1aad-b5b8-4d25-b1a0-511900d33167 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b58b530 49de05a5-3f1e-44bf-9847-3b5971b2b008 /tidb/cdc/default/default/changefeed/info/57cbb323-f9fa-474b-be02-12d72c8e6d75 {"upstream-id":7374683623901191057,"namespace":"default","changefeed-id":"57cbb323-f9fa-474b-be02-12d72c8e6d75","sink-uri":"pulsar://127.0.0.1:6650/ticdc-processor-resolved-ts-fallback-test-15758?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T14:59:11.269795445+08:00","start-ts":450114971518107656,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114971570798593} /tidb/cdc/default/default/changefeed/status/57cbb323-f9fa-474b-be02-12d72c8e6d75 {"checkpoint-ts":450114973182984195,"min-table-barrier-ts":450114973182984195,"admin-job-type":0} /tidb/cdc/default/default/task/position/49de05a5-3f1e-44bf-9847-3b5971b2b008/57cbb323-f9fa-474b-be02-12d72c8e6d75 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/c68f1aad-b5b8-4d25-b1a0-511900d33167/57cbb323-f9fa-474b-be02-12d72c8e6d75 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683623901191057 {"id":7374683623901191057,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/57cbb323-f9fa-474b-be02-12d72c8e6d75 {UpstreamID:7374683623901191057 Namespace:default ID:57cbb323-f9fa-474b-be02-12d72c8e6d75 SinkURI:pulsar://127.0.0.1:6650/ticdc-processor-resolved-ts-fallback-test-15758?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 14:59:11.269795445 +0800 CST StartTs:450114971518107656 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0038d01b0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114971570798593} {CheckpointTs:450114973182984195 MinTableBarrierTs:450114973182984195 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/49de05a5-3f1e-44bf-9847-3b5971b2b008 {"id":"49de05a5-3f1e-44bf-9847-3b5971b2b008","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/c68f1aad-b5b8-4d25-b1a0-511900d33167 {"id":"c68f1aad-b5b8-4d25-b1a0-511900d33167","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b58b4bb c68f1aad-b5b8-4d25-b1a0-511900d33167 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b58b530 49de05a5-3f1e-44bf-9847-3b5971b2b008 /tidb/cdc/default/default/changefeed/info/57cbb323-f9fa-474b-be02-12d72c8e6d75 {"upstream-id":7374683623901191057,"namespace":"default","changefeed-id":"57cbb323-f9fa-474b-be02-12d72c8e6d75","sink-uri":"pulsar://127.0.0.1:6650/ticdc-processor-resolved-ts-fallback-test-15758?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T14:59:11.269795445+08:00","start-ts":450114971518107656,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114971570798593} /tidb/cdc/default/default/changefeed/status/57cbb323-f9fa-474b-be02-12d72c8e6d75 {"checkpoint-ts":450114973182984195,"min-table-barrier-ts":450114973182984195,"admin-job-type":0} /tidb/cdc/default/default/task/position/49de05a5-3f1e-44bf-9847-3b5971b2b008/57cbb323-f9fa-474b-be02-12d72c8e6d75 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/c68f1aad-b5b8-4d25-b1a0-511900d33167/57cb+ grep -q 'failed to get info:' b323-f9fa-474b-be02-12d72c8e6d75 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683623901191057 {"id":7374683623901191057,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/57cbb323-f9fa-474b-be02-12d72c8e6d75 {UpstreamID:7374683623901191057 Namespace:default ID:57cbb323-f9fa-474b-be02-12d72c8e6d75 SinkURI:pulsar://127.0.0.1:6650/ticdc-processor-resolved-ts-fallback-test-15758?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 14:59:11.269795445 +0800 CST StartTs:450114971518107656 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0038d01b0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114971570798593} {CheckpointTs:450114973182984195 MinTableBarrierTs:450114973182984195 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/49de05a5-3f1e-44bf-9847-3b5971b2b008 {"id":"49de05a5-3f1e-44bf-9847-3b5971b2b008","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/c68f1aad-b5b8-4d25-b1a0-511900d33167 {"id":"c68f1aad-b5b8-4d25-b1a0-511900d33167","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b58b4bb c68f1aad-b5b8-4d25-b1a0-511900d33167 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b58b530 49de05a5-3f1e-44bf-9847-3b5971b2b008 /tidb/cdc/default/default/changefeed/info/57cbb323-f9fa-474b-be02-12d72c8e6d75 {"upstream-id":7374683623901191057,"namespace":"default","changefeed-id":"57cbb323-f9fa-474b-be02-12d72c8e6d75","sink-uri":"pulsar://127.0.0.1:6650/ticdc-processor-resolved-ts-fallback-test-15758?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T14:59:11.269795445+08:00","start-ts":450114971518107656,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114971570798593} /tidb/cdc/default/default/changefeed/status/57cbb323-f9fa-474b-be02-12d72c8e6d75 {"checkpoint-ts":450114973182984195,"min-table-barrier-ts":450114973182984195,"admin-job-type":0} /tidb/cdc/default/default/task/position/49de05a5-3f1e-44bf-9847-3b5971b2b008/57cbb323-f9fa-474b-be02-12d72c8e6d75 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/c68f1aad-b5b8-4d25-b1a0-511900d33167/57cbb323-f9fa-474b-be02-12d72c8e6d75 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683623901191057 {"id":7374683623901191057,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Thu May 30 14:59:17 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/InjectChangefeedDDLError=return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.57265728.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] // withEnv [Pipeline] // podTemplate [Pipeline] // podTemplate table ddl_manager.finish_mark not exists for 29-th check, retry later [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] // stage [Pipeline] // withEnv [Pipeline] // withEnv [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] // stage [Pipeline] // stage [Pipeline] } [Pipeline] } check diff failed 2-th time, retry later + set +x [Thu May 30 14:59:18 CST 2024] <<<<<< START Pulsar consumer in resolve_lock case >>>>>> go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20240510070134-e7f530c64790 go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20240527072219-a97e6464c01d go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading github.com/pingcap/kvproto v0.0.0-20240522024016-df42997c2c57 go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20240527072219-a97e6464c01d go: downloading github.com/tikv/pd/client v0.0.0-20240210135946-3488a653ddd9 go: downloading github.com/BurntSushi/toml v1.3.2 go: downloading go.uber.org/zap v1.27.0 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading go.uber.org/atomic v1.11.0 go: downloading github.com/pingcap/failpoint v0.0.0-20240412033321-fd0796e60f86 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/google/uuid v1.6.0 go: downloading google.golang.org/grpc v1.64.0 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/google/btree v1.1.2 go: downloading github.com/golang/protobuf v1.5.4 go: downloading github.com/prometheus/client_golang v1.19.1 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading golang.org/x/sync v0.7.0 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 go: downloading google.golang.org/protobuf v1.34.1 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/prometheus/common v0.53.0 go: downloading github.com/prometheus/procfs v0.15.0 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading golang.org/x/sys v0.20.0 check diff successfully table processor_resolved_ts_fallback.t2 not exists for 1-th check, retry later go: downloading golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842 go: downloading github.com/cloudfoundry/gosigar v1.3.6 wait process cdc.test exit for 1-th time... go: downloading golang.org/x/net v0.25.0 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240521202816-d264139d666e go: downloading golang.org/x/text v0.15.0 table ddl_manager.finish_mark not exists for 30-th check, retry later wait process cdc.test exit for 2-th time... check diff failed 3-th time, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 14:59:20 CST 2024] <<<<<< run test case event_filter success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:20 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/1f31080e-dea6-452a-928f-f2520071d53b {UpstreamID:7374683618857110233 Namespace:default ID:1f31080e-dea6-452a-928f-f2520071d53b SinkURI:pulsar://127.0.0.1:6650/ticdc-owner-resign-test-21608?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 14:59:10.376799563 +0800 CST StartTs:450114971293450241 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003de1710 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114971332509699} {CheckpointTs:450114973705175042 MinTableBarrierTs:450114973705175042 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/50a49725-4bf2-4bad-a255-bee45960c0a9 {"id":"50a49725-4bf2-4bad-a255-bee45960c0a9","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/b3bf1dab-31b1-436b-af8a-0040bb261270 {"id":"b3bf1dab-31b1-436b-af8a-0040bb261270","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b52e1bd b3bf1dab-31b1-436b-af8a-0040bb261270 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b52e259 50a49725-4bf2-4bad-a255-bee45960c0a9 /tidb/cdc/default/default/changefeed/info/1f31080e-dea6-452a-928f-f2520071d53b {"upstream-id":7374683618857110233,"namespace":"default","changefeed-id":"1f31080e-dea6-452a-928f-f2520071d53b","sink-uri":"pulsar://127.0.0.1:6650/ticdc-owner-resign-test-21608?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T14:59:10.376799563+08:00","start-ts":450114971293450241,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114971332509699} /tidb/cdc/default/default/changefeed/status/1f31080e-dea6-452a-928f-f2520071d53b {"checkpoint-ts":450114973705175042,"min-table-barrier-ts":450114973705175042,"admin-job-type":0} /tidb/cdc/default/default/task/position/50a49725-4bf2-4bad-a255-bee45960c0a9/1f31080e-dea6-452a-928f-f2520071d53b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b3bf1dab-31b1-436b-af8a-0040bb261270/1f31080e-dea6-452a-928f-f2520071d53b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683618857110233 {"id":7374683618857110233,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/1f31080e-dea6-452a-928f-f2520071d53b {UpstreamID:7374683618857110233 Namespace:default ID:1f31080e-dea6-452a-928f-f2520071d53b SinkURI:pulsar://127.0.0.1:6650/ticdc-owner-resign-test-21608?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 14:59:10.376799563 +0800 CST StartTs:450114971293450241 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003de1710 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114971332509699} {CheckpointTs:450114973705175042 MinTableBarrierTs:450114973705175042 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/50a49725-4bf2-4bad-a255-bee45960c0a9 {"id":"50a49725-4bf2-4bad-a255-bee45960c0a9","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/b3bf1dab-31b1-436b-af8a-0040bb261270 {"id":"b3bf1dab-31b1-436b-af8a-0040bb261270","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b52e1bd b3bf1dab-31b1-436b-af8a-0040bb261270 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b52e259 50a49725-4bf2-4bad-a255-bee45960c0a9 /tidb/cdc/default/default/changefeed/info/1f31080e-dea6-452a-928f-f2520071d53b {"upstream-id":7374683618857110233,"namespace":"default","changefeed-id":"1f31080e-dea6-452a-928f-f2520071d53b","sink-uri":"pulsar://127.0.0.1:6650/ticdc-owner-resign-test-21608?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T14:59:10.376799563+08:00","start-ts":450114971293450241,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114971332509699} /tidb/cdc/default/default/changefeed/status/1f31080e-dea6-452a-928f-f2520071d53b {"checkpoint-ts":450114973705175042,"min-table-barrier-ts":450114973705175042,"admin-job-type":0} /tidb/cdc/default/default/task/position/50a49725-4bf2-4bad-a255-bee45960c0a9/1f31080e-dea6-452a-928f-f2520071d53b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b3bf1dab-31b1-436b-af8a-0040bb261270/1f31080e-dea6-452a-928f-f2520071d53b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683618857110233 {"id":7374683618857110233,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/1f31080e-dea6-452a-928f-f2520071d53b {UpstreamID:7374683618857110233 Namespace:default ID:1f31080e-dea6-452a-928f-f2520071d53b SinkURI:pulsar://127.0.0.1:6650/ticdc-owner-resign-test-21608?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 14:59:10.376799563 +0800 CST StartTs:450114971293450241 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003de1710 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450114971332509699} {CheckpointTs:450114973705175042 MinTableBarrierTs:450114973705175042 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/50a49725-4bf2-4bad-a255-bee45960c0a9 {"id":"50a49725-4bf2-4bad-a255-bee45960c0a9","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/b3bf1dab-31b1-436b-af8a-0040bb261270 {"id":"b3bf1dab-31b1-436b-af8a-0040bb261270","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b52e1bd b3bf1dab-31b1-436b-af8a-0040bb261270 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b52e259 50a49725-4bf2-4bad-a255-bee45960c0a9 /tidb/cdc/default/default/changefeed/info/1f31080e-dea6-452a-928f-f2520071d53b {"upstream-id":7374683618857110233,"namespace":"default","changefeed-id":"1f31080e-dea6-452a-928f-f2520071d53b","sink-uri":"pulsar://127.0.0.1:6650/ticdc-owner-resign-test-21608?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T14:59:10.376799563+08:00","start-ts":450114971293450241,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450114971332509699} /tidb/cdc/default/default/changefeed/status/1f31080e-dea6-452a-928f-f2520071d53b {"checkpoint-ts":450114973705175042,"min-table-barrier-ts":450114973705175042,"admin-job-type":0} /tidb/cdc/default/default/task/position/50a49725-4bf2-4bad-a255-bee45960c0a9/1f31080e-dea6-452a-928f-f2520071d53b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b3bf1dab-31b1-436b-af8a-0040bb261270/1f31080e-dea6-452a-928f-f2520071d53b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374683618857110233 {"id":7374683618857110233,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^2$' 2 run task successfully % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 [2024/05/30 14:59:17.229 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001954b40/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 14:59:19.764 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001954b40/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] {"level":"warn","ts":"2024-05-30T14:59:20.468206+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f2afc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} {"level":"warn","ts":"2024-05-30T14:59:20.46871+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00084f180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} {"level":"warn","ts":"2024-05-30T14:59:20.56814+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000e5c540/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} [2024/05/30 14:59:20.834 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001b5c1e0/127.0.0.1:2379] [attempt=0] [error="rpc error: code = NotFound desc = etcdserver: requested lease not found"] [2024/05/30 14:59:20.836 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0019b61e0/127.0.0.1:2479] [attempt=0] [error="rpc error: code = NotFound desc = etcdserver: requested lease not found"] [Thu May 30 14:59:20 CST 2024] <<<<<< START cdc server in batch_add_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + GO_FAILPOINTS= + etcd_info_msg='etcd info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.26492651.out server --log-file /tmp/tidb_cdc_test/batch_add_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/batch_add_table/cdc_data --cluster-id default + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:20 GMT < Content-Length: 681 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e659a6a7-31f9-4fb8-bf8c-724b4b3a762a {"id":"e659a6a7-31f9-4fb8-bf8c-724b4b3a762a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318fc84af9e6f7 e659a6a7-31f9-4fb8-bf8c-724b4b3a762a /tidb/cdc/default/default/upstream/7374683520812005767 {"id":7374683520812005767,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e659a6a7-31f9-4fb8-bf8c-724b4b3a762a {"id":"e659a6a7-31f9-4fb8-bf8c-724b4b3a762a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318fc84af9e6f7 e659a6a7-31f9-4fb8-bf8c-724b4b3a762a /tidb/cdc/default/default/upstream/7374683520812005767 {"id":7374683520812005767,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e659a6a7-31f9-4fb8-bf8c-724b4b3a762a {"id":"e659a6a7-31f9-4fb8-bf8c-724b4b3a762a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318fc84af9e6f7 e659a6a7-31f9-4fb8-bf8c-724b4b3a762a /tidb/cdc/default/default/upstream/7374683520812005767 {"id":7374683520812005767,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5769.out cli changefeed create --start-ts=450114964225523713 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-sink-retry-test-18481?protocol=canal-json&enable-tidb-extension=true' -c changefeed-error-1 table processor_resolved_ts_fallback.t2 exists table processor_resolved_ts_fallback.t3 not exists for 1-th check, retry later go: downloading github.com/pingcap/tipb v0.0.0-20240507090649-2bf6bb0cb996 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/shirou/gopsutil/v3 v3.24.4 go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/gorilla/mux v1.8.0 go: downloading github.com/danjacques/gofslock v0.0.0-20220131014315-6e321f4509c8 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/cockroachdb/pebble v0.0.0-20220415182917-06c9d3be25b3 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/aws/aws-sdk-go v1.44.259 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.1 go: downloading github.com/klauspost/compress v1.17.8 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/iancoleman/strcase v0.2.0 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/stretchr/testify v1.9.0 go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/xitongsys/parquet-go v1.6.0 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading golang.org/x/time v0.5.0 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/sasha-s/go-deadlock v0.3.1 Create changefeed successfully! ID: changefeed-error-1 Info: {"upstream_id":7374683520812005767,"namespace":"default","id":"changefeed-error-1","sink_uri":"pulsar://127.0.0.1:6650/ticdc-sink-retry-test-18481?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T14:59:21.448806452+08:00","start_ts":450114964225523713,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114964225523713,"checkpoint_ts":450114964225523713,"checkpoint_time":"2024-05-30 14:58:43.248"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading github.com/fatih/color v1.16.0 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.6 go: downloading golang.org/x/oauth2 v0.18.0 go: downloading google.golang.org/api v0.170.0 go: downloading github.com/spf13/cobra v1.8.0 go: downloading github.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0 go: downloading github.com/vbauerster/mpb/v7 v7.5.3 go: downloading golang.org/x/term v0.20.0 go: downloading github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/pingcap/badger v1.5.1-0.20230103063557-828f39b09b6d go: downloading github.com/dolthub/maphash v0.1.0 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/Masterminds/semver v1.5.0 go: downloading k8s.io/api v0.28.6 go: downloading github.com/emirpasic/gods v1.18.1 go: downloading github.com/petermattis/goid v0.0.0-20231207134359-e60b3f734c67 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/mattn/go-runewidth v0.0.15 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading cloud.google.com/go v0.112.2 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.1 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.1 go: downloading golang.org/x/crypto v0.23.0 go: downloading github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/rogpeppe/go-internal v1.12.0 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading github.com/golang-jwt/jwt/v5 v5.2.0 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac table cdc_tiflash_test.multi_data_type exists check diff failed 1-th time, retry later go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da check diff failed 4-th time, retry later table ddl_manager.finish_mark not exists for 31-th check, retry later go: downloading github.com/go-logr/logr v1.4.1 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/DataDog/zstd v1.5.5 go: downloading k8s.io/apimachinery v0.28.6 go: downloading k8s.io/klog/v2 v2.120.1 go: downloading github.com/google/gofuzz v1.2.0 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd table processor_resolved_ts_fallback.t3 not exists for 2-th check, retry later + set +x check_changefeed_status 127.0.0.1:8300 changefeed-error-1 warning last_warning ErrExecDDLFailed + endpoint=127.0.0.1:8300 + changefeed_id=changefeed-error-1 + expected_state=warning + field=last_warning + error_pattern=ErrExecDDLFailed ++ curl 127.0.0.1:8300/api/v2/changefeeds/changefeed-error-1/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 86 100 86 0 0 602 0 --:--:-- --:--:-- --:--:-- 605 + info='{"state":"normal","resolved_ts":450114964225523713,"checkpoint_ts":450114964225523713}' + echo '{"state":"normal","resolved_ts":450114964225523713,"checkpoint_ts":450114964225523713}' {"state":"normal","resolved_ts":450114964225523713,"checkpoint_ts":450114964225523713} ++ echo '{"state":"normal","resolved_ts":450114964225523713,"checkpoint_ts":450114964225523713}' ++ jq -r .state + state=normal + [[ ! normal == \w\a\r\n\i\n\g ]] + echo 'changefeed state normal does not equal to warning' changefeed state normal does not equal to warning + exit 1 run task failed 1-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) \033[0;36m<<< Run all test success >>>\033[0m + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:23 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f0d2a8e5-0463-4386-b19a-d7da89f28748 {"id":"f0d2a8e5-0463-4386-b19a-d7da89f28748","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b79d216 f0d2a8e5-0463-4386-b19a-d7da89f28748 /tidb/cdc/default/default/upstream/7374683651579560194 {"id":7374683651579560194,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f0d2a8e5-0463-4386-b19a-d7da89f28748 {"id":"f0d2a8e5-0463-4386-b19a-d7da89f28748","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b79d216 f0d2a8e5-0463-4386-b19a-d7da89f28748 /tidb/cdc/default/default/upstream/7374683651579560194 {"id":7374683651579560194,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f0d2a8e5-0463-4386-b19a-d7da89f28748 {"id":"f0d2a8e5-0463-4386-b19a-d7da89f28748","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b79d216 f0d2a8e5-0463-4386-b19a-d7da89f28748 /tidb/cdc/default/default/upstream/7374683651579560194 {"id":7374683651579560194,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.cli.2692.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-batch-add-table-test-16381?protocol=canal-json&enable-tidb-extension=true' [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc already exists) check diff failed 2-th time, retry later [Pipeline] // cache Create changefeed successfully! ID: 2aa683b3-a3c8-4b50-b58b-a2c37a465631 Info: {"upstream_id":7374683651579560194,"namespace":"default","id":"2aa683b3-a3c8-4b50-b58b-a2c37a465631","sink_uri":"pulsar://127.0.0.1:6650/ticdc-batch-add-table-test-16381?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T14:59:24.148950211+08:00","start_ts":450114974880890894,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114974880890894,"checkpoint_ts":450114974880890894,"checkpoint_time":"2024-05-30 14:59:23.895"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... check diff failed 1-th time, retry later [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage INSERT - Takes(s): 10.0, Count: 1764, OPS: 176.5, Avg(us): 5313, Min(us): 1563, Max(us): 96496, 95th(us): 6000, 99th(us): 86000 [Pipeline] } table ddl_manager.finish_mark not exists for 32-th check, retry later [Pipeline] // container [Pipeline] } check diff failed 5-th time, retry later [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } check_changefeed_status 127.0.0.1:8300 changefeed-error-1 warning last_warning ErrExecDDLFailed + endpoint=127.0.0.1:8300 + changefeed_id=changefeed-error-1 + expected_state=warning + field=last_warning + error_pattern=ErrExecDDLFailed ++ curl 127.0.0.1:8300/api/v2/changefeeds/changefeed-error-1/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed [Pipeline] // podTemplate + set +x [Thu May 30 14:59:25 CST 2024] <<<<<< START Pulsar consumer in batch_add_table case >>>>>> [Pipeline] } 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 253 100 253 0 0 1745 0 --:--:-- --:--:-- --:--:-- 1744 100 253 100 253 0 0 1743 0 --:--:-- --:--:-- --:--:-- 1744 + info='{"state":"warning","resolved_ts":450114964776288281,"checkpoint_ts":450114964776288281,"last_warning":{"time":"2024-05-30T14:59:23.414248832+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}}' + echo '{"state":"warning","resolved_ts":450114964776288281,"checkpoint_ts":450114964776288281,"last_warning":{"time":"2024-05-30T14:59:23.414248832+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}}' {"state":"warning","resolved_ts":450114964776288281,"checkpoint_ts":450114964776288281,"last_warning":{"time":"2024-05-30T14:59:23.414248832+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}} ++ echo '{"state":"warning","resolved_ts":450114964776288281,"checkpoint_ts":450114964776288281,"last_warning":{"time":"2024-05-30T14:59:23.414248832+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec' DDL 'failed"}}' ++ jq -r .state + state=warning + [[ ! warning == \w\a\r\n\i\n\g ]] + [[ -z last_warning ]] ++ echo '{"state":"warning","resolved_ts":450114964776288281,"checkpoint_ts":450114964776288281,"last_warning":{"time":"2024-05-30T14:59:23.414248832+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec' DDL 'failed"}}' ++ jq -r .last_warning.message + error_msg='[CDC:ErrExecDDLFailed]exec DDL failed' + [[ ! [CDC:ErrExecDDLFailed]exec DDL failed =~ ErrExecDDLFailed ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5832.out cli changefeed remove -c changefeed-error-1 table processor_resolved_ts_fallback.t3 exists cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^2$' [Pipeline] // withEnv 2 run task successfully check diff successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table batch_add_table.finish_mark not exists for 1-th check, retry later [Pipeline] } table ddl_manager.finish_mark not exists for 33-th check, retry later Changefeed remove successfully. ID: changefeed-error-1 CheckpointTs: 450114964776288281 SinkURI: pulsar://127.0.0.1:6650/ticdc-sink-retry-test-18481?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... go: downloading github.com/jmespath/go-jmespath v0.4.0 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading github.com/google/s2a-go v0.1.7 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading github.com/felixge/httpsnoop v1.0.4 wait process cdc.test exit for 1-th time... check diff failed 2-th time, retry later [Pipeline] // stage check diff successfully {"level":"warn","ts":"2024-05-30T14:59:26.834478+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00084f180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} {"level":"warn","ts":"2024-05-30T14:59:26.83458+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f2afc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} [Pipeline] } cdc.test: no process found wait process cdc.test exit for 2-th time... process cdc.test already exit [Thu May 30 14:59:27 CST 2024] <<<<<< run test case tiflash success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) killing cdc1: 2598 cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' Error: Get "http://127.0.0.1:8301/api/v2/captures": read tcp 127.0.0.1:55284->127.0.0.1:8301: read: connection reset by peer run task failed 1-th time, retry later + set +x wait process cdc.test exit for 1-th time... check diff successfully % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 table batch_add_table.finish_mark not exists for 2-th check, retry later check diff failed 1-th time, retry later wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... table ddl_manager.finish_mark not exists for 34-th check, retry later cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused run task failed 2-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Thu May 30 14:59:29 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/txnutil/gc/InjectActualGCSafePoint=return(9223372036854775807)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.58955897.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table batch_add_table.finish_mark exists check diff successfully check diff failed 2-th time, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } wait process cdc.test exit for 1-th time... [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } wait process cdc.test exit for 2-th time... [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } table ddl_manager.finish_mark not exists for 35-th check, retry later [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } wait process cdc.test exit for 3-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 14:59:32 CST 2024] <<<<<< run test case batch_add_table success! >>>>>> check diff failed 3-th time, retry later [2024/05/30 14:59:33.120 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0019b61e0/127.0.0.1:2479] [attempt=0] [error="rpc error: code = NotFound desc = etcdserver: requested lease not found"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:32 GMT < Content-Length: 681 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bc092e0e-3da4-449a-82d6-acbf0691f6ab {"id":"bc092e0e-3da4-449a-82d6-acbf0691f6ab","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/22318fc84af9e770 bc092e0e-3da4-449a-82d6-acbf0691f6ab /tidb/cdc/default/default/upstream/7374683520812005767 {"id":7374683520812005767,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bc092e0e-3da4-449a-82d6-acbf0691f6ab {"id":"bc092e0e-3da4-449a-82d6-acbf0691f6ab","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/22318fc84af9e770 bc092e0e-3da4-449a-82d6-acbf0691f6ab /tidb/cdc/default/default/upstream/7374683520812005767 {"id":7374683520812005767,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bc092e0e-3da4-449a-82d6-acbf0691f6ab {"id":"bc092e0e-3da4-449a-82d6-acbf0691f6ab","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/22318fc84af9e770 bc092e0e-3da4-449a-82d6-acbf0691f6ab /tidb/cdc/default/default/upstream/7374683520812005767 {"id":7374683520812005767,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5940.out cli changefeed create --start-ts=450114964225523713 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-sink-retry-test-18481?protocol=canal-json&enable-tidb-extension=true' -c changefeed-error-2 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Create changefeed successfully! ID: changefeed-error-2 Info: {"upstream_id":7374683520812005767,"namespace":"default","id":"changefeed-error-2","sink_uri":"pulsar://127.0.0.1:6650/ticdc-sink-retry-test-18481?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T14:59:33.462717371+08:00","start_ts":450114964225523713,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114964225523713,"checkpoint_ts":450114964225523713,"checkpoint_time":"2024-05-30 14:58:43.248"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused run task failed 3-th time, retry later table ddl_manager.finish_mark not exists for 36-th check, retry later INSERT - Takes(s): 20.0, Count: 2872, OPS: 143.7, Avg(us): 6547, Min(us): 1563, Max(us): 96496, 95th(us): 69000, 99th(us): 86000 + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-error-2 failed [CDC:ErrSnapshotLostByGC] + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error-2 + expected_state=failed + error_msg='[CDC:ErrSnapshotLostByGC]' + tls_dir='[CDC:ErrSnapshotLostByGC]' + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error-2 -s check diff failed 4-th time, retry later + info='{ "upstream_id": 7374683520812005767, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 0, "checkpoint_time": "0001-01-01 00:00:00.000", "error": { "time": "2024-05-30T14:59:33.473500205+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 450114964225523713 is earlier than or equal to GC safepoint at 9223372036854775807" } }' + echo '{ "upstream_id": 7374683520812005767, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 0, "checkpoint_time": "0001-01-01 00:00:00.000", "error": { "time": "2024-05-30T14:59:33.473500205+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 450114964225523713 is earlier than or equal to GC safepoint at 9223372036854775807" } }' { "upstream_id": 7374683520812005767, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 0, "checkpoint_time": "0001-01-01 00:00:00.000", "error": { "time": "2024-05-30T14:59:33.473500205+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 450114964225523713 is earlier than or equal to GC safepoint at 9223372036854775807" } } ++ jq -r .state ++ echo '{' '"upstream_id":' 7374683520812005767, '"namespace":' '"default",' '"id":' '"changefeed-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 0, '"checkpoint_time":' '"0001-01-01' '00:00:00.000",' '"error":' '{' '"time":' '"2024-05-30T14:59:33.473500205+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSnapshotLostByGC",' '"message":' '"[CDC:ErrSnapshotLostByGC]fail' to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 450114964225523713 is earlier than or equal to GC safepoint at '9223372036854775807"' '}' '}' + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ jq -r .error.message ++ echo '{' '"upstream_id":' 7374683520812005767, '"namespace":' '"default",' '"id":' '"changefeed-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 0, '"checkpoint_time":' '"0001-01-01' '00:00:00.000",' '"error":' '{' '"time":' '"2024-05-30T14:59:33.473500205+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSnapshotLostByGC",' '"message":' '"[CDC:ErrSnapshotLostByGC]fail' to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 450114964225523713 is earlier than or equal to GC safepoint at '9223372036854775807"' '}' '}' + message='[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 450114964225523713 is earlier than or equal to GC safepoint at 9223372036854775807' + [[ ! [CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 450114964225523713 is earlier than or equal to GC safepoint at 9223372036854775807 =~ \[CDC:ErrSnapshotLostByGC] ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6005.out cli changefeed remove -c changefeed-error-2 table ddl_manager.finish_mark not exists for 37-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Changefeed remove successfully. ID: changefeed-error-2 CheckpointTs: 0 SinkURI: pulsar://127.0.0.1:6650/ticdc-sink-retry-test-18481?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... check diff failed 5-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/region_merge/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table ddl_manager.finish_mark not exists for 38-th check, retry later + set +x wait process cdc.test exit for 1-th time... check diff successfully wait process cdc.test exit for 2-th time... wait process cdc.test exit for 1-th time... wait process cdc.test exit for 3-th time... check diff successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/05/30 14:59:38.979 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001954b40/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 14:59:39 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/redo/ChangefeedNewRedoManagerError=2*return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.60446046.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 39-th check, retry later wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 14:59:40 CST 2024] <<<<<< run test case owner_resign success! >>>>>> start tidb cluster in /tmp/tidb_cdc_test/region_merge Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' 1 run task successfully check diff failed 1-th time, retry later check diff failed 1-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 40-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/multi_rocks/run.sh using Sink-Type: pulsar... <<================= + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 06:59:42 GMT < Content-Length: 681 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/640ffa50-511c-4a55-ba94-75e991d062db {"id":"640ffa50-511c-4a55-ba94-75e991d062db","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 6 /tidb/cdc/default/__cdc_meta__/owner/22318fc84af9e7be 640ffa50-511c-4a55-ba94-75e991d062db /tidb/cdc/default/default/upstream/7374683520812005767 {"id":7374683520812005767,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/640ffa50-511c-4a55-ba94-75e991d062db {"id":"640ffa50-511c-4a55-ba94-75e991d062db","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 6 /tidb/cdc/default/__cdc_meta__/owner/22318fc84af9e7be 640ffa50-511c-4a55-ba94-75e991d062db /tidb/cdc/default/default/upstream/7374683520812005767 {"id":7374683520812005767,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/640ffa50-511c-4a55-ba94-75e991d062db {"id":"640ffa50-511c-4a55-ba94-75e991d062db","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 6 /tidb/cdc/default/__cdc_meta__/owner/22318fc84af9e7be 640ffa50-511c-4a55-ba94-75e991d062db /tidb/cdc/default/default/upstream/7374683520812005767 {"id":7374683520812005767,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6089.out cli changefeed create --start-ts=0 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-sink-retry-test-18481?protocol=canal-json&enable-tidb-extension=true' -c changefeed-initialize-error check diff successfully check diff failed 2-th time, retry later Create changefeed successfully! ID: changefeed-initialize-error Info: {"upstream_id":7374683520812005767,"namespace":"default","id":"changefeed-initialize-error","sink_uri":"pulsar://127.0.0.1:6650/ticdc-sink-retry-test-18481?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T14:59:43.200839584+08:00","start_ts":450114979901997063,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114979901997063,"checkpoint_ts":450114979901997063,"checkpoint_time":"2024-05-30 14:59:43.049"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 41-th check, retry later INSERT - Takes(s): 30.0, Count: 4278, OPS: 142.6, Avg(us): 6609, Min(us): 1538, Max(us): 96496, 95th(us): 69000, 99th(us): 85000 wait process cdc.test exit for 3-th time... + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7374683520812005767, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 450114979901997063, "checkpoint_time": "2024-05-30 14:59:43.049", "error": { "time": "2024-05-30T14:59:43.374224472+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' + echo '{ "upstream_id": 7374683520812005767, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 450114979901997063, "checkpoint_time": "2024-05-30 14:59:43.049", "error": { "time": "2024-05-30T14:59:43.374224472+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' { "upstream_id": 7374683520812005767, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 450114979901997063, "checkpoint_time": "2024-05-30 14:59:43.049", "error": { "time": "2024-05-30T14:59:43.374224472+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } } ++ echo '{' '"upstream_id":' 7374683520812005767, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"warning",' '"checkpoint_tso":' 450114979901997063, '"checkpoint_time":' '"2024-05-30' '14:59:43.049",' '"error":' '{' '"time":' '"2024-05-30T14:59:43.374224472+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .state + state=warning + [[ ! warning == \n\o\r\m\a\l ]] + echo 'changefeed state warning does not equal to normal' changefeed state warning does not equal to normal + exit 1 run task failed 1-th time, retry later wait process cdc.test exit for 4-th time... check diff failed 3-th time, retry later cdc.test: no process found wait process cdc.test exit for 5-th time... process cdc.test already exit [Thu May 30 14:59:45 CST 2024] <<<<<< run test case processor_resolved_ts_fallback success! >>>>>> \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc already exists) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [Pipeline] // cache [Pipeline] } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 42-th check, retry later [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [2024/05/30 14:59:46.283 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001954b40/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 14:59:46.323 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0015d2780/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 4-th time, retry later check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7374683520812005767, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 450114979901997063, "checkpoint_time": "2024-05-30 14:59:43.049", "error": { "time": "2024-05-30T14:59:43.374224472+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' + echo '{ "upstream_id": 7374683520812005767, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 450114979901997063, "checkpoint_time": "2024-05-30 14:59:43.049", "error": { "time": "2024-05-30T14:59:43.374224472+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' { "upstream_id": 7374683520812005767, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 450114979901997063, "checkpoint_time": "2024-05-30 14:59:43.049", "error": { "time": "2024-05-30T14:59:43.374224472+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } } ++ echo '{' '"upstream_id":' 7374683520812005767, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"warning",' '"checkpoint_tso":' 450114979901997063, '"checkpoint_time":' '"2024-05-30' '14:59:43.049",' '"error":' '{' '"time":' '"2024-05-30T14:59:43.374224472+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .state + state=warning + [[ ! warning == \n\o\r\m\a\l ]] + echo 'changefeed state warning does not equal to normal' changefeed state warning does not equal to normal + exit 1 run task failed 2-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 43-th check, retry later Run finished, takes 34.149359631s INSERT - Takes(s): 34.1, Count: 5000, OPS: 146.4, Avg(us): 6432, Min(us): 1538, Max(us): 96496, 95th(us): 66000, 99th(us): 84000 [Thu May 30 14:59:48 CST 2024] <<<<<< START Pulsar consumer in big_txn case >>>>>> table big_txn.usertable not exists for 1-th check, retry later [2024/05/30 14:59:48.674 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001954b40/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 14:59:48.744 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0015d2780/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 5-th time, retry later table ddl_manager.finish_mark not exists for 44-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/processor_etcd_worker_delay/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table big_txn.usertable exists [2024/05/30 14:59:51.226 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0015d2780/127.0.0.1:2379] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [2024/05/30 14:59:51.275 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc001954b40/127.0.0.1:2479] [attempt=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7374683520812005767, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 450114981776326665, "checkpoint_time": "2024-05-30 14:59:50.199", "error": null }' + echo '{ "upstream_id": 7374683520812005767, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 450114981776326665, "checkpoint_time": "2024-05-30 14:59:50.199", "error": null }' { "upstream_id": 7374683520812005767, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 450114981776326665, "checkpoint_time": "2024-05-30 14:59:50.199", "error": null } ++ echo '{' '"upstream_id":' 7374683520812005767, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 450114981776326665, '"checkpoint_time":' '"2024-05-30' '14:59:50.199",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374683520812005767, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 450114981776326665, '"checkpoint_time":' '"2024-05-30' '14:59:50.199",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6236.out cli changefeed pause -c changefeed-initialize-error [2024/05/30 14:59:51.587 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc0019b61e0/127.0.0.1:2479] [attempt=0] [error="rpc error: code = NotFound desc = etcdserver: requested lease not found"] check diff failed 6-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 45-th check, retry later PASS ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) coverage: 1.9% of statements in github.com/pingcap/tiflow/... start tidb cluster in /tmp/tidb_cdc_test/processor_etcd_worker_delay Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error stopped changefeed new redo manager injected error + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=stopped + error_msg=changefeed + tls_dir=error + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s check diff successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 46-th check, retry later + info='{ "upstream_id": 7374683520812005767, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 450114982038470662, "checkpoint_time": "2024-05-30 14:59:51.199", "error": { "time": "2024-05-30T14:59:43.374224472+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' + echo '{ "upstream_id": 7374683520812005767, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 450114982038470662, "checkpoint_time": "2024-05-30 14:59:51.199", "error": { "time": "2024-05-30T14:59:43.374224472+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' { "upstream_id": 7374683520812005767, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 450114982038470662, "checkpoint_time": "2024-05-30 14:59:51.199", "error": { "time": "2024-05-30T14:59:43.374224472+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } } ++ echo '{' '"upstream_id":' 7374683520812005767, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"stopped",' '"checkpoint_tso":' 450114982038470662, '"checkpoint_time":' '"2024-05-30' '14:59:51.199",' '"error":' '{' '"time":' '"2024-05-30T14:59:43.374224472+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7374683520812005767, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"stopped",' '"checkpoint_tso":' 450114982038470662, '"checkpoint_time":' '"2024-05-30' '14:59:51.199",' '"error":' '{' '"time":' '"2024-05-30T14:59:43.374224472+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .error.message + message='changefeed new redo manager injected error' + [[ ! changefeed new redo manager injected error =~ changefeed ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6305.out cli changefeed resume -c changefeed-initialize-error ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... wait process cdc.test exit for 2-th time... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 14:59:54 CST 2024] <<<<<< run test case changefeed_pause_resume success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213120d80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-n5fjf, pid:3828, start at 2024-05-30 14:59:53.57019027 +0800 CST m=+6.563052408 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:53.579 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:53.577 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:53.577 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213120d80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-n5fjf, pid:3828, start at 2024-05-30 14:59:53.57019027 +0800 CST m=+6.563052408 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:53.579 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:53.577 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:53.577 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2131230c0015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-n5fjf, pid:3910, start at 2024-05-30 14:59:53.715585916 +0800 CST m=+6.633543272 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:53.722 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:53.717 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:53.717 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/region_merge/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/region_merge/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7374683520812005767, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 450114982038470662, "checkpoint_time": "2024-05-30 14:59:51.199", "error": null }' + echo '{ "upstream_id": 7374683520812005767, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 450114982038470662, "checkpoint_time": "2024-05-30 14:59:51.199", "error": null }' { "upstream_id": 7374683520812005767, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 450114982038470662, "checkpoint_time": "2024-05-30 14:59:51.199", "error": null } ++ echo '{' '"upstream_id":' 7374683520812005767, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 450114982038470662, '"checkpoint_time":' '"2024-05-30' '14:59:51.199",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374683520812005767, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 450114982038470662, '"checkpoint_time":' '"2024-05-30' '14:59:51.199",' '"error":' null '}' ++ jq -r .error.message [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } table ddl_manager.finish_mark not exists for 47-th check, retry later [Pipeline] // container + message=null + [[ ! null =~ null ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6365.out cli changefeed remove -c changefeed-initialize-error [Pipeline] } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21312dfc0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8f2lb, pid:1215, start at 2024-05-30 14:59:54.415624537 +0800 CST m=+48.998138408 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:54.423 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:54.417 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:54.417 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21312dfc0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8f2lb, pid:1215, start at 2024-05-30 14:59:54.415624537 +0800 CST m=+48.998138408 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:54.423 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:54.417 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:54.417 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213131040009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8f2lb, pid:1301, start at 2024-05-30 14:59:54.570717355 +0800 CST m=+49.086208794 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:01:54.578 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-14:59:54.561 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:49:54.561 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } Changefeed remove successfully. ID: changefeed-initialize-error CheckpointTs: 450114982038470662 SinkURI: pulsar://127.0.0.1:6650/ticdc-sink-retry-test-18481?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... [Thu May 30 14:59:57 CST 2024] <<<<<< START cdc server in region_merge case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.region_merge.52305232.out server --log-file /tmp/tidb_cdc_test/region_merge/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/region_merge/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 48-th check, retry later + set +x wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_session_done_during_task.cli.2866.out cli tso query --pd=http://127.0.0.1:2379 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:00:00 CST 2024] <<<<<< run test case changefeed_error success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:00:00 GMT < Content-Length: 657 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0f210008-0f8f-46aa-aa01-fd69d08b44b8 {"id":"0f210008-0f8f-46aa-aa01-fd69d08b44b8","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c1c5f35 0f210008-0f8f-46aa-aa01-fd69d08b44b8 /tidb/cdc/default/default/upstream/7374683830591982696 {"id":7374683830591982696,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0f210008-0f8f-46aa-aa01-fd69d08b44b8 {"id":"0f210008-0f8f-46aa-aa01-fd69d08b44b8","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c1c5f35 0f210008-0f8f-46aa-aa01-fd69d08b44b8 /tidb/cdc/default/default/upstream/7374683830591982696 {"id":7374683830591982696,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0f210008-0f8f-46aa-aa01-fd69d08b44b8 {"id":"0f210008-0f8f-46aa-aa01-fd69d08b44b8","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c1c5f35 0f210008-0f8f-46aa-aa01-fd69d08b44b8 /tidb/cdc/default/default/upstream/7374683830591982696 {"id":7374683830591982696,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.region_merge.cli.5280.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-region-merge-test-14644?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: 255af448-1351-418f-b25f-8c3c01b71322 Info: {"upstream_id":7374683830591982696,"namespace":"default","id":"255af448-1351-418f-b25f-8c3c01b71322","sink_uri":"pulsar://127.0.0.1:6650/ticdc-region-merge-test-14644?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T15:00:01.009190453+08:00","start_ts":450114984562393091,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114984562393091,"checkpoint_ts":450114984562393091,"checkpoint_time":"2024-05-30 15:00:00.827"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x + tso='450114984258306049 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114984258306049 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 15:00:01 CST 2024] <<<<<< START cdc server in capture_session_done_during_task case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorManagerHandleNewChangefeedDelay=sleep(2000)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_session_done_during_task.29042906.out server --log-file /tmp/tidb_cdc_test/capture_session_done_during_task/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/capture_session_done_during_task/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x [Thu May 30 15:00:02 CST 2024] <<<<<< START Pulsar consumer in region_merge case >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2131aac00017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-k5r94, pid:4081, start at 2024-05-30 15:00:02.383854429 +0800 CST m=+5.243986104 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:02.391 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:02.352 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:02.352 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2131aac00017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-k5r94, pid:4081, start at 2024-05-30 15:00:02.383854429 +0800 CST m=+5.243986104 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:02.391 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:02.352 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:02.352 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2131ac080004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-k5r94, pid:4173, start at 2024-05-30 15:00:02.439328766 +0800 CST m=+5.251423535 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:02.446 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:02.434 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:02.434 +0800 All versions after safe point can be accessed. (DO NOT EDIT) table ddl_manager.finish_mark not exists for 49-th check, retry later Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } split_and_random_merge scale: 20 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:00:04 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/384db301-c0db-48ca-95a4-4064c6268f02 {"id":"384db301-c0db-48ca-95a4-4064c6268f02","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b74708e 384db301-c0db-48ca-95a4-4064c6268f02 /tidb/cdc/default/default/upstream/7374683655178674454 {"id":7374683655178674454,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/384db301-c0db-48ca-95a4-4064c6268f02 {"id":"384db301-c0db-48ca-95a4-4064c6268f02","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b74708e 384db301-c0db-48ca-95a4-4064c6268f02 /tidb/cdc/default/default/upstream/7374683655178674454 {"id":7374683655178674454,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/384db301-c0db-48ca-95a4-4064c6268f02 {"id":"384db301-c0db-48ca-95a4-4064c6268f02","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84b74708e 384db301-c0db-48ca-95a4-4064c6268f02 /tidb/cdc/default/default/upstream/7374683655178674454 {"id":7374683655178674454,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table ddl_manager.finish_mark not exists for 50-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/cli/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... [Thu May 30 15:00:05 CST 2024] <<<<<< START cdc server in processor_etcd_worker_delay case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/orchestrator/ProcessorEtcdDelay=return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_etcd_worker_delay.54695471.out server --log-file /tmp/tidb_cdc_test/processor_etcd_worker_delay/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_etcd_worker_delay/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 51-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/cli Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:00:08 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/780d3391-7dc6-4348-b62c-90a9f5221599 {"id":"780d3391-7dc6-4348-b62c-90a9f5221599","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c4654b7 780d3391-7dc6-4348-b62c-90a9f5221599 /tidb/cdc/default/default/upstream/7374683879559505791 {"id":7374683879559505791,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/780d3391-7dc6-4348-b62c-90a9f5221599 {"id":"780d3391-7dc6-4348-b62c-90a9f5221599","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c4654b7 780d3391-7dc6-4348-b62c-90a9f5221599 /tidb/cdc/default/default/upstream/7374683879559505791 {"id":7374683879559505791,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/780d3391-7dc6-4348-b62c-90a9f5221599 {"id":"780d3391-7dc6-4348-b62c-90a9f5221599","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c4654b7 780d3391-7dc6-4348-b62c-90a9f5221599 /tidb/cdc/default/default/upstream/7374683879559505791 {"id":7374683879559505791,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Thu May 30 15:00:08 CST 2024] <<<<<< START Pulsar consumer in processor_etcd_worker_delay case >>>>>> table ddl_manager.finish_mark not exists for 52-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release *************************** 1. row *************************** count(distinct region_id): 1 table ddl_manager.finish_mark not exists for 53-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Thu May 30 15:00:11 CST 2024] <<<<<< START Pulsar consumer in capture_session_done_during_task case >>>>>> lease 22318fc84b74708e revoked table capture_session_done_during_task.t exists check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 54-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later split_and_random_merge scale: 40 table ddl_manager.finish_mark not exists for 55-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/ddl_sequence/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table processor_delay.t1 exists table processor_delay.t2 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 56-th check, retry later check diff successfully check diff failed 1-th time, retry later table processor_delay.t2 exists table processor_delay.t3 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21328f40000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:9727, start at 2024-05-30 15:00:16.990377713 +0800 CST m=+5.589649230 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:16.998 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:16.976 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:16.976 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/ddl_sequence Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... check diff successfully table ddl_manager.finish_mark not exists for 57-th check, retry later table processor_delay.t3 exists table processor_delay.t4 not exists for 1-th check, retry later wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21328f40000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:9727, start at 2024-05-30 15:00:16.990377713 +0800 CST m=+5.589649230 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:16.998 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:16.976 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:16.976 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21328f38000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:9777, start at 2024-05-30 15:00:16.989331459 +0800 CST m=+5.504812908 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:16.997 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:16.974 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:16.974 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cli/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cli/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cli/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/cli/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/cli/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:00:20 CST 2024] <<<<<< run test case capture_session_done_during_task success! >>>>>> table ddl_manager.finish_mark not exists for 58-th check, retry later *************************** 1. row *************************** count(distinct region_id): 27 table processor_delay.t4 exists table processor_delay.t5 not exists for 1-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11073.out cli tso query --pd=http://127.0.0.1:2379 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table processor_delay.t5 exists table processor_delay.t6 not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 59-th check, retry later + set +x + tso='450114990145536004 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114990145536004 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 15:00:23 CST 2024] <<<<<< START cdc server in cli case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.1111111113.out server --log-file /tmp/tidb_cdc_test/cli/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cli/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 60-th check, retry later table processor_delay.t6 exists table processor_delay.t7 not exists for 1-th check, retry later split_and_random_merge scale: 80 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:00:26 GMT < Content-Length: 657 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c5009667-6010-443e-83ca-8b2827cd830a {"id":"c5009667-6010-443e-83ca-8b2827cd830a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c7dda44 c5009667-6010-443e-83ca-8b2827cd830a /tidb/cdc/default/default/upstream/7374683938475784960 {"id":7374683938475784960,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c5009667-6010-443e-83ca-8b2827cd830a {"id":"c5009667-6010-443e-83ca-8b2827cd830a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c7dda44 c5009667-6010-443e-83ca-8b2827cd830a /tidb/cdc/default/default/upstream/7374683938475784960 {"id":7374683938475784960,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c5009667-6010-443e-83ca-8b2827cd830a {"id":"c5009667-6010-443e-83ca-8b2827cd830a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84c7dda44 c5009667-6010-443e-83ca-8b2827cd830a /tidb/cdc/default/default/upstream/7374683938475784960 {"id":7374683938475784960,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11157.out cli changefeed create --start-ts=450114990145536004 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-cli-test-11188?protocol=canal-json&enable-tidb-extension=true' --tz=Asia/Shanghai -c=custom-changefeed-name [WARN] --tz is deprecated in changefeed settings. table ddl_manager.finish_mark not exists for 61-th check, retry later table processor_delay.t7 exists table processor_delay.t8 not exists for 1-th check, retry later Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7374683938475784960,"namespace":"default","id":"custom-changefeed-name","sink_uri":"pulsar://127.0.0.1:6650/ticdc-cli-test-11188?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T15:00:27.416055256+08:00","start_ts":450114990145536004,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114990145536004,"checkpoint_ts":450114990145536004,"checkpoint_time":"2024-05-30 15:00:22.125"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x [Thu May 30 15:00:28 CST 2024] <<<<<< START Pulsar consumer in cli case >>>>>> table test.simple not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 62-th check, retry later table processor_delay.t8 exists table processor_delay.t9 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/ddl_attributes/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table ddl_manager.finish_mark not exists for 63-th check, retry later table processor_delay.t9 not exists for 2-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213361f40014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-rbkd5, pid:7330, start at 2024-05-30 15:00:30.505133874 +0800 CST m=+6.377273338 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:30.512 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:30.511 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:30.511 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.simple exists table test.`simple-dash` not exists for 1-th check, retry later table processor_delay.t9 exists table processor_delay.t10 exists table processor_delay.t11 not exists for 1-th check, retry later table test.`simple-dash` exists + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7374683938475784960, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 450114990565228556, "checkpoint_time": "2024-05-30 15:00:23.726", "error": null }' + echo '{ "upstream_id": 7374683938475784960, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 450114990565228556, "checkpoint_time": "2024-05-30 15:00:23.726", "error": null }' { "upstream_id": 7374683938475784960, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 450114990565228556, "checkpoint_time": "2024-05-30 15:00:23.726", "error": null } ++ echo '{' '"upstream_id":' 7374683938475784960, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 450114990565228556, '"checkpoint_time":' '"2024-05-30' '15:00:23.726",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374683938475784960, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 450114990565228556, '"checkpoint_time":' '"2024-05-30' '15:00:23.726",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] changefeed count 1 check pass, pd_addr: http://127.0.0.1:2379 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213361f40014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-rbkd5, pid:7330, start at 2024-05-30 15:00:30.505133874 +0800 CST m=+6.377273338 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:30.512 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:30.511 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:30.511 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213363c80014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-rbkd5, pid:7407, start at 2024-05-30 15:00:30.612838838 +0800 CST m=+6.383809014 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:30.620 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:30.628 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:30.628 +0800 All versions after safe point can be accessed. (DO NOT EDIT) changefeed count 1 check pass, pd_addr: http://127.0.0.1:2679 Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_sequence/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_sequence/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table ddl_manager.finish_mark not exists for 64-th check, retry later changefeed count 1 check pass, pd_addr: http://127.0.0.1:2779 start tidb cluster in /tmp/tidb_cdc_test/ddl_attributes Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... changefeed count 1 check pass, pd_addr: http://127.0.0.1:2379,http://127.0.0.1:2679,http://127.0.0.1:2779 Error: [CDC:ErrChangefeedUpdateRefused]changefeed update error: can only update changefeed config when it is stopped or failed update changefeed config should fail when changefeed is running, got Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config SyncPointInterval] From: To:0xc00336ff50} {Type:update Path:[Config SyncPointRetention] From: To:0xc00336ff58} {Type:update Path:[Config Consistent] From: To:0xc0009376c0} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11466.out cli changefeed --changefeed-id custom-changefeed-name pause PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark exists check diff successfully table processor_delay.t11 exists table processor_delay.t12 not exists for 1-th check, retry later wait process cdc.test exit for 1-th time... + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.cli.8636.out cli tso query --pd=http://127.0.0.1:2379 + set +x wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... Verifying downstream PD is started... table processor_delay.t12 exists table processor_delay.t13 not exists for 1-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:00:37 CST 2024] <<<<<< run test case ddl_manager success! >>>>>> Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + set +x + tso='450114993772560385 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + awk -F ' ' '{print $1}' + echo 450114993772560385 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + set +x [Thu May 30 15:00:37 CST 2024] <<<<<< START cdc server in ddl_sequence case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.86708672.out server --log-file /tmp/tidb_cdc_test/ddl_sequence/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_sequence/cdc_data --cluster-id default * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t13 exists table processor_delay.t14 not exists for 1-th check, retry later Starting Upstream TiDB... + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=stopped + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7374683938475784960, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 450114993383276551, "checkpoint_time": "2024-05-30 15:00:34.476", "error": null }' + echo '{ "upstream_id": 7374683938475784960, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 450114993383276551, "checkpoint_time": "2024-05-30 15:00:34.476", "error": null }' { "upstream_id": 7374683938475784960, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 450114993383276551, "checkpoint_time": "2024-05-30 15:00:34.476", "error": null } ++ echo '{' '"upstream_id":' 7374683938475784960, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 450114993383276551, '"checkpoint_time":' '"2024-05-30' '15:00:34.476",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7374683938475784960, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 450114993383276551, '"checkpoint_time":' '"2024-05-30' '15:00:34.476",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11535.out cli changefeed update --pd=http://127.0.0.1:2379,http://127.0.0.1:2679,http://127.0.0.1:2779 --config=/tmp/tidb_cdc_test/cli/changefeed.toml --no-confirm --changefeed-id custom-changefeed-name Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config SyncPointInterval] From: To:0xc0039ceeb8} {Type:update Path:[Config SyncPointRetention] From: To:0xc0039ceec8} {Type:update Path:[Config Consistent] From: To:0xc000d942a0} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} Update changefeed config successfully! ID: custom-changefeed-name Info: {"upstream_id":7374683938475784960,"namespace":"default","id":"custom-changefeed-name","sink_uri":"pulsar://127.0.0.1:6650/ticdc-cli-test-11188?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T15:00:27.416055256+08:00","start_ts":450114990145536004,"admin_job_type":1,"config":{"memory_quota":1073741824,"case_sensitive":true,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","encoder_concurrency":32,"terminator":"\r\n","enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"stopped","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114993383276551,"checkpoint_ts":450114993383276551,"checkpoint_time":"2024-05-30 15:00:34.476"} PASS coverage: 2.8% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:00:40 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1f5ef675-7f96-408e-bd26-af68f1d11ba7 {"id":"1f5ef675-7f96-408e-bd26-af68f1d11ba7","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ca7d6c3 1f5ef675-7f96-408e-bd26-af68f1d11ba7 /tidb/cdc/default/default/upstream/7374683997701142389 {"id":7374683997701142389,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1f5ef675-7f96-408e-bd26-af68f1d11ba7 {"id":"1f5ef675-7f96-408e-bd26-af68f1d11ba7","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ca7d6c3 1f5ef675-7f96-408e-bd26-af68f1d11ba7 /tidb/cdc/default/default/upstream/7374683997701142389 {"id":7374683997701142389,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1f5ef675-7f96-408e-bd26-af68f1d11ba7 {"id":"1f5ef675-7f96-408e-bd26-af68f1d11ba7","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ca7d6c3 1f5ef675-7f96-408e-bd26-af68f1d11ba7 /tidb/cdc/default/default/upstream/7374683997701142389 {"id":7374683997701142389,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.cli.8718.out cli changefeed create --start-ts=450114993772560385 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-ddl-sequence-test-22082?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: fe242e9a-981b-488f-825c-cf8dad71d4ef Info: {"upstream_id":7374683997701142389,"namespace":"default","id":"fe242e9a-981b-488f-825c-cf8dad71d4ef","sink_uri":"pulsar://127.0.0.1:6650/ticdc-ddl-sequence-test-22082?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T15:00:41.191325302+08:00","start_ts":450114993772560385,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114993772560385,"checkpoint_ts":450114993772560385,"checkpoint_time":"2024-05-30 15:00:35.961"} PASS table processor_delay.t14 exists table processor_delay.t15 not exists for 1-th check, retry later coverage: 2.4% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11570.out cli changefeed --changefeed-id custom-changefeed-name resume PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + set +x [Thu May 30 15:00:42 CST 2024] <<<<<< START Pulsar consumer in ddl_sequence case >>>>>> table processor_delay.t15 exists table processor_delay.t16 not exists for 1-th check, retry later + set +x ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_sequence.finish_mark not exists for 1-th check, retry later table processor_delay.t16 exists table processor_delay.t17 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213444dc0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8f2lb, pid:4178, start at 2024-05-30 15:00:45.010216897 +0800 CST m=+5.146868660 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:45.017 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:44.983 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:44.983 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213444dc0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8f2lb, pid:4178, start at 2024-05-30 15:00:45.010216897 +0800 CST m=+5.146868660 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:45.017 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:44.983 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:44.983 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213446600007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-8f2lb, pid:4264, start at 2024-05-30 15:00:45.088744465 +0800 CST m=+5.143424846 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:02:45.095 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:00:45.080 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:50:45.080 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_attributes/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_attributes/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7374683938475784960, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 450114996529004553, "checkpoint_time": "2024-05-30 15:00:46.476", "error": null }' + echo '{ "upstream_id": 7374683938475784960, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 450114996529004553, "checkpoint_time": "2024-05-30 15:00:46.476", "error": null }' { "upstream_id": 7374683938475784960, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 450114996529004553, "checkpoint_time": "2024-05-30 15:00:46.476", "error": null } ++ echo '{' '"upstream_id":' 7374683938475784960, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 450114996529004553, '"checkpoint_time":' '"2024-05-30' '15:00:46.476",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374683938475784960, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 450114996529004553, '"checkpoint_time":' '"2024-05-30' '15:00:46.476",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11637.out cli changefeed --changefeed-id custom-changefeed-name remove table processor_delay.t17 exists table processor_delay.t18 not exists for 1-th check, retry later table ddl_sequence.finish_mark not exists for 2-th check, retry later Changefeed remove successfully. ID: custom-changefeed-name CheckpointTs: 450114996529004553 SinkURI: pulsar://127.0.0.1:6650/ticdc-cli-test-11188?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.cli.5522.out cli tso query --pd=http://127.0.0.1:2379 table processor_delay.t18 exists table processor_delay.t19 not exists for 1-th check, retry later + set +x table ddl_sequence.finish_mark not exists for 3-th check, retry later + set +x + tso='450114997068496897 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450114997068496897 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 15:00:50 CST 2024] <<<<<< START cdc server in ddl_attributes case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.55515553.out server --log-file /tmp/tidb_cdc_test/ddl_attributes/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_attributes/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t19 exists table processor_delay.t20 not exists for 1-th check, retry later table ddl_sequence.finish_mark not exists for 4-th check, retry later changefeed count 0 check pass, pd_addr: http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11704.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-cli-test-11188?protocol=canal-json&enable-tidb-extension=true' --tz=Asia/Shanghai -c=custom-changefeed-name [WARN] --tz is deprecated in changefeed settings. Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7374683938475784960,"namespace":"default","id":"custom-changefeed-name","sink_uri":"pulsar://127.0.0.1:6650/ticdc-cli-test-11188?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T15:00:52.92282331+08:00","start_ts":450114998180511752,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114998180511752,"checkpoint_ts":450114998180511752,"checkpoint_time":"2024-05-30 15:00:52.776"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:00:53 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0d1a1907-8469-4f18-ab2b-08de10cb4e5f {"id":"0d1a1907-8469-4f18-ab2b-08de10cb4e5f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ce53bc2 0d1a1907-8469-4f18-ab2b-08de10cb4e5f /tidb/cdc/default/default/upstream/7374684054539673203 {"id":7374684054539673203,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0d1a1907-8469-4f18-ab2b-08de10cb4e5f {"id":"0d1a1907-8469-4f18-ab2b-08de10cb4e5f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ce53bc2 0d1a1907-8469-4f18-ab2b-08de10cb4e5f /tidb/cdc/default/default/upstream/7374684054539673203 {"id":7374684054539673203,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0d1a1907-8469-4f18-ab2b-08de10cb4e5f {"id":"0d1a1907-8469-4f18-ab2b-08de10cb4e5f","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ce53bc2 0d1a1907-8469-4f18-ab2b-08de10cb4e5f /tidb/cdc/default/default/upstream/7374684054539673203 {"id":7374684054539673203,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.cli.5596.out cli changefeed create --start-ts=450114997068496897 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-ddl-attributes-test-13667?protocol=canal-json&enable-tidb-extension=true' table ddl_sequence.finish_mark not exists for 5-th check, retry later Create changefeed successfully! ID: 04d18b2f-aa06-4331-a7c4-e93416c18ea0 Info: {"upstream_id":7374684054539673203,"namespace":"default","id":"04d18b2f-aa06-4331-a7c4-e93416c18ea0","sink_uri":"pulsar://127.0.0.1:6650/ticdc-ddl-attributes-test-13667?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T15:00:53.585877643+08:00","start_ts":450114997068496897,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450114997068496897,"checkpoint_ts":450114997068496897,"checkpoint_time":"2024-05-30 15:00:48.534"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table processor_delay.t20 exists table processor_delay.t21 not exists for 1-th check, retry later + set +x + set +x [Thu May 30 15:00:55 CST 2024] <<<<<< START Pulsar consumer in ddl_attributes case >>>>>> table ddl_sequence.finish_mark exists table processor_delay.t21 exists table processor_delay.t22 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/multi_source/run.sh using Sink-Type: pulsar... <<================= table processor_delay.t22 exists table processor_delay.t23 not exists for 1-th check, retry later + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7374683938475784960, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 450114999163551745, "checkpoint_time": "2024-05-30 15:00:56.526", "error": null }' + echo '{ "upstream_id": 7374683938475784960, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 450114999163551745, "checkpoint_time": "2024-05-30 15:00:56.526", "error": null }' { "upstream_id": 7374683938475784960, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 450114999163551745, "checkpoint_time": "2024-05-30 15:00:56.526", "error": null } ++ echo '{' '"upstream_id":' 7374683938475784960, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 450114999163551745, '"checkpoint_time":' '"2024-05-30' '15:00:56.526",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374683938475784960, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 450114999163551745, '"checkpoint_time":' '"2024-05-30' '15:00:56.526",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] check diff successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11794.out cli unsafe delete-service-gc-safepoint wait process cdc.test exit for 1-th time... table big_txn.usertable1 not exists for 1-th check, retry later table big_txn.usertable1 not exists for 2-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Confirm that you know what this command will do and use it at your own risk [Y/N] CDC service GC safepoint truncated in PD! PASS Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } coverage: 1.9% of statements in github.com/pingcap/tiflow/... [Pipeline] // timeout [Pipeline] } wait process cdc.test exit for 2-th time... [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } wait process cdc.test exit for 3-th time... [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } table big_txn.usertable1 not exists for 3-th check, retry later [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } table processor_delay.t23 exists table processor_delay.t24 not exists for 1-th check, retry later [Pipeline] // stage [Pipeline] } cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:00:59 CST 2024] <<<<<< run test case ddl_sequence success! >>>>>> + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11819.out cli unsafe reset --no-confirm --pd=http://127.0.0.1:2379 reset and all metadata truncated in PD! PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... table big_txn.usertable1 not exists for 4-th check, retry later table processor_delay.t24 not exists for 2-th check, retry later table ddl_attributes.attributes_t1_new not exists for 1-th check, retry later + set +x table ddl_attributes.attributes_t1_new not exists for 2-th check, retry later table big_txn.usertable1 not exists for 5-th check, retry later table processor_delay.t24 exists table processor_delay.t25 exists table processor_delay.t26 not exists for 1-th check, retry later table ddl_attributes.attributes_t1_new exists table ddl_attributes.finish_mark not exists for 1-th check, retry later + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11899.out cli unsafe resolve-lock --region=22 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... table big_txn.usertable1 not exists for 6-th check, retry later table processor_delay.t26 exists table processor_delay.t27 not exists for 1-th check, retry later + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11923.out cli unsafe resolve-lock --region=22 --ts=450115000526700551 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... table ddl_attributes.finish_mark not exists for 2-th check, retry later table big_txn.usertable1 not exists for 7-th check, retry later table processor_delay.t27 exists table processor_delay.t28 not exists for 1-th check, retry later + set +x % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 218 100 212 100 6 158k 4601 --:--:-- --:--:-- --:--:-- 207k { "error_msg": "[CDC:ErrAPIInvalidParam]invalid log level: json: cannot unmarshal string into Go value of type struct { Level string \"json:\\\"log_level\\\"\" }", "error_code": "CDC:ErrAPIInvalidParam" table ddl_attributes.finish_mark not exists for 3-th check, retry later table big_txn.usertable1 not exists for 8-th check, retry later table processor_delay.t28 exists table processor_delay.t29 not exists for 1-th check, retry later TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 1-th time, retry later check data successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 15:01:07 CST 2024] <<<<<< run test case ddl_puller_lag success! >>>>>> table big_txn.usertable1 exists } % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 176 100 176 0 0 208k 0 --:--:-- --:--:-- --:--:-- 171k { "version": "v7.5.1-58-g8f92142d7", "git_hash": "8f92142d7874bfad2a8db6ca100604c383fca9df", "id": "97bfda19-34fa-4fc5-b7e3-b86c10d76321", "pid": 11116, "is_owner": true table ddl_attributes.finish_mark not exists for 4-th check, retry later table processor_delay.t29 exists table processor_delay.t30 not exists for 1-th check, retry later }wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 15:01:12 CST 2024] <<<<<< run test case cli success! >>>>>> table ddl_attributes.finish_mark not exists for 5-th check, retry later table processor_delay.t30 exists table processor_delay.t31 not exists for 1-th check, retry later table ddl_attributes.finish_mark not exists for 6-th check, retry later table processor_delay.t31 exists table processor_delay.t32 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/ddl_only_block_related_table/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table ddl_attributes.finish_mark not exists for 7-th check, retry later table processor_delay.t32 exists table processor_delay.t33 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/resourcecontrol/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/resourcecontrol Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... table ddl_attributes.finish_mark not exists for 8-th check, retry later table processor_delay.t33 exists table processor_delay.t34 not exists for 1-th check, retry later table ddl_attributes.finish_mark not exists for 9-th check, retry later table processor_delay.t34 exists table processor_delay.t35 not exists for 1-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/savepoint/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table ddl_attributes.finish_mark exists check diff successfully table processor_delay.t35 exists table processor_delay.t36 not exists for 1-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... start tidb cluster in /tmp/tidb_cdc_test/ddl_only_block_related_table Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Verifying downstream PD is started... wait process cdc.test exit for 3-th time... start tidb cluster in /tmp/tidb_cdc_test/savepoint Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit table processor_delay.t36 exists table processor_delay.t37 not exists for 1-th check, retry later Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [Thu May 30 15:01:26 CST 2024] <<<<<< run test case ddl_attributes success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... table processor_delay.t37 exists table processor_delay.t38 not exists for 1-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table processor_delay.t38 exists table processor_delay.t39 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t39 exists table processor_delay.t40 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2136fdd00017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-rbkd5, pid:9805, start at 2024-05-30 15:01:29.633132801 +0800 CST m=+5.592940651 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:03:29.641 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:01:29.638 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:51:29.638 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2136fdd00017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-rbkd5, pid:9805, start at 2024-05-30 15:01:29.633132801 +0800 CST m=+5.592940651 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:03:29.641 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:01:29.638 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:51:29.638 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2136ff640006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-rbkd5, pid:9876, start at 2024-05-30 15:01:29.69628456 +0800 CST m=+5.573542325 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:03:29.704 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:01:29.689 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:51:29.689 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/resourcecontrol/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/resourcecontrol/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t40 exists table processor_delay.t41 not exists for 1-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.cli.11144.out cli tso query --pd=http://127.0.0.1:2379 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21373c480010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-mwr1g, pid:6535, start at 2024-05-30 15:01:33.606981658 +0800 CST m=+5.517979069 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:03:33.615 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:01:33.586 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:51:33.586 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21373c480010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-mwr1g, pid:6535, start at 2024-05-30 15:01:33.606981658 +0800 CST m=+5.517979069 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:03:33.615 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:01:33.586 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:51:33.586 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21373ddc000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-mwr1g, pid:6585, start at 2024-05-30 15:01:33.704824912 +0800 CST m=+5.532536647 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:03:33.714 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:01:33.687 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:51:33.687 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t41 exists table processor_delay.t42 not exists for 1-th check, retry later + set +x + tso='450115009207074817 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115009207074817 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 15:01:36 CST 2024] <<<<<< START cdc server in resourcecontrol case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.1117611178.out server --log-file /tmp/tidb_cdc_test/resourcecontrol/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/resourcecontrol/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21375a680014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:12874, start at 2024-05-30 15:01:35.541398376 +0800 CST m=+5.150389336 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:03:35.549 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:01:35.514 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:51:35.514 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21375a680014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:12874, start at 2024-05-30 15:01:35.541398376 +0800 CST m=+5.150389336 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:03:35.549 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:01:35.514 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:51:35.514 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f21375ca80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:12955, start at 2024-05-30 15:01:35.67649329 +0800 CST m=+5.232776299 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:03:35.683 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:01:35.658 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:51:35.658 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/savepoint/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/savepoint/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Thu May 30 15:01:36 CST 2024] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.78317833.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t42 exists table processor_delay.t43 not exists for 1-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.cli.14210.out cli tso query --pd=http://127.0.0.1:2379 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:01:39 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e0a3374d-528a-433c-b09a-d1508baddffb {"id":"e0a3374d-528a-433c-b09a-d1508baddffb","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84d93e1c2 e0a3374d-528a-433c-b09a-d1508baddffb /tidb/cdc/default/default/upstream/7374684248221012313 {"id":7374684248221012313,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e0a3374d-528a-433c-b09a-d1508baddffb {"id":"e0a3374d-528a-433c-b09a-d1508baddffb","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84d93e1c2 e0a3374d-528a-433c-b09a-d1508baddffb /tidb/cdc/default/default/upstream/7374684248221012313 {"id":7374684248221012313,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e0a3374d-528a-433c-b09a-d1508baddffb {"id":"e0a3374d-528a-433c-b09a-d1508baddffb","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84d93e1c2 e0a3374d-528a-433c-b09a-d1508baddffb /tidb/cdc/default/default/upstream/7374684248221012313 {"id":7374684248221012313,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.cli.11220.out cli changefeed create --start-ts=450115009207074817 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-resourcecontrol-test-19048?protocol=canal-json&enable-tidb-extension=true' table processor_delay.t43 exists table processor_delay.t44 not exists for 1-th check, retry later < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:01:39 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/95afb6f7-6118-4da1-886f-898a82b59135 {"id":"95afb6f7-6118-4da1-886f-898a82b59135","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84da767bc 95afb6f7-6118-4da1-886f-898a82b59135 /tidb/cdc/default/default/upstream/7374684260771247141 {"id":7374684260771247141,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/95afb6f7-6118-4da1-886f-898a82b59135 {"id":"95afb6f7-6118-4da1-886f-898a82b59135","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84da767bc 95afb6f7-6118-4da1-886f-898a82b59135 /tidb/cdc/default/default/upstream/7374684260771247141 {"id":7374684260771247141,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/95afb6f7-6118-4da1-886f-898a82b59135 {"id":"95afb6f7-6118-4da1-886f-898a82b59135","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84da767bc 95afb6f7-6118-4da1-886f-898a82b59135 /tidb/cdc/default/default/upstream/7374684260771247141 {"id":7374684260771247141,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.cli.7878.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-common-1-test-8611?protocol=canal-json&enable-tidb-extension=true' -c=ddl-only-block-related-table Create changefeed successfully! ID: 8deaa640-94a1-4d18-9a2f-bcb1e57bcd92 Info: {"upstream_id":7374684248221012313,"namespace":"default","id":"8deaa640-94a1-4d18-9a2f-bcb1e57bcd92","sink_uri":"pulsar://127.0.0.1:6650/ticdc-resourcecontrol-test-19048?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T15:01:39.920672743+08:00","start_ts":450115009207074817,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115009207074817,"checkpoint_ts":450115009207074817,"checkpoint_time":"2024-05-30 15:01:34.839"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... Create changefeed successfully! ID: ddl-only-block-related-table Info: {"upstream_id":7374684260771247141,"namespace":"default","id":"ddl-only-block-related-table","sink_uri":"pulsar://127.0.0.1:6650/ticdc-common-1-test-8611?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T15:01:40.057427955+08:00","start_ts":450115010530115586,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115010530115586,"checkpoint_ts":450115010530115586,"checkpoint_time":"2024-05-30 15:01:39.886"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x + tso='450115010314633217 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115010314633217 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Thu May 30 15:01:40 CST 2024] <<<<<< START cdc server in savepoint case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.1423614238.out server --log-file /tmp/tidb_cdc_test/savepoint/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/savepoint/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir + set +x [Thu May 30 15:01:41 CST 2024] <<<<<< START Pulsar consumer in resourcecontrol case >>>>>> [Pipeline] } [Pipeline] // timeout [Pipeline] } + set +x [Thu May 30 15:01:41 CST 2024] <<<<<< START Pulsar consumer in ddl_only_block_related_table case >>>>>> [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv table processor_delay.t44 exists table processor_delay.t45 not exists for 1-th check, retry later table ddl_only_block_related_table.finish_mark not exists for 1-th check, retry later [Pipeline] } [Pipeline] // stage [Pipeline] } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:01:43 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/674f2171-25e9-4d98-8904-797f85842666 {"id":"674f2171-25e9-4d98-8904-797f85842666","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84dae8bc4 674f2171-25e9-4d98-8904-797f85842666 /tidb/cdc/default/default/upstream/7374684268984367247 {"id":7374684268984367247,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/674f2171-25e9-4d98-8904-797f85842666 {"id":"674f2171-25e9-4d98-8904-797f85842666","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84dae8bc4 674f2171-25e9-4d98-8904-797f85842666 /tidb/cdc/default/default/upstream/7374684268984367247 {"id":7374684268984367247,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/674f2171-25e9-4d98-8904-797f85842666 {"id":"674f2171-25e9-4d98-8904-797f85842666","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84dae8bc4 674f2171-25e9-4d98-8904-797f85842666 /tidb/cdc/default/default/upstream/7374684268984367247 {"id":7374684268984367247,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.cli.14283.out cli changefeed create --start-ts=450115010314633217 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-savepoint-test-30834?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: f6bc4f70-a4eb-41db-92be-e98cfbb96903 Info: {"upstream_id":7374684268984367247,"namespace":"default","id":"f6bc4f70-a4eb-41db-92be-e98cfbb96903","sink_uri":"pulsar://127.0.0.1:6650/ticdc-savepoint-test-30834?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T15:01:44.080601518+08:00","start_ts":450115010314633217,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115010314633217,"checkpoint_ts":450115010314633217,"checkpoint_time":"2024-05-30 15:01:39.064"} PASS table processor_delay.t45 exists table processor_delay.t46 not exists for 1-th check, retry later table ddl_only_block_related_table.finish_mark not exists for 2-th check, retry later coverage: 2.4% of statements in github.com/pingcap/tiflow/... table resourcecontrol.finish_mark not exists for 1-th check, retry later + set +x [Thu May 30 15:01:45 CST 2024] <<<<<< START Pulsar consumer in savepoint case >>>>>> table processor_delay.t46 exists table processor_delay.t47 not exists for 1-th check, retry later table ddl_only_block_related_table.finish_mark not exists for 3-th check, retry later table resourcecontrol.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... table savepoint.finish_mark not exists for 1-th check, retry later wait process cdc.test exit for 3-th time... table processor_delay.t47 exists table processor_delay.t48 not exists for 1-th check, retry later [2024/05/30 15:01:44.514 +08:00] [INFO] [pd_service_discovery.go:631] ["[pd] switch leader"] [new-leader=http://127.0.0.1:2379] [old-leader=] [2024/05/30 15:01:44.515 +08:00] [INFO] [pd_service_discovery.go:197] ["[pd] init cluster id"] [cluster-id=7374683640738619107] [2024/05/30 15:01:44.515 +08:00] [INFO] [client.go:600] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2024/05/30 15:01:44.515 +08:00] [INFO] [tso_client.go:231] ["[tso] switch dc tso global allocator serving address"] [dc-location=global] [new-address=http://127.0.0.1:2379] [2024/05/30 15:01:44.515 +08:00] [INFO] [tso_dispatcher.go:323] ["[tso] tso dispatcher created"] [dc-location=global] [2024/05/30 15:01:44.515 +08:00] [INFO] [client.go:648] ["[pd] service mode changed"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2024/05/30 15:01:44.517 +08:00] [INFO] [pd_service_discovery.go:631] ["[pd] switch leader"] [new-leader=http://127.0.0.1:2379] [old-leader=] [2024/05/30 15:01:44.517 +08:00] [INFO] [pd_service_discovery.go:197] ["[pd] init cluster id"] [cluster-id=7374683640738619107] [2024/05/30 15:01:44.517 +08:00] [INFO] [client.go:600] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2024/05/30 15:01:44.517 +08:00] [INFO] [tso_client.go:231] ["[tso] switch dc tso global allocator serving address"] [dc-location=global] [new-address=http://127.0.0.1:2379] [2024/05/30 15:01:44.517 +08:00] [INFO] [tso_dispatcher.go:323] ["[tso] tso dispatcher created"] [dc-location=global] [2024/05/30 15:01:44.517 +08:00] [INFO] [client.go:648] ["[pd] service mode changed"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2024/05/30 15:01:44.518 +08:00] [INFO] [tikv_driver.go:200] ["using API V1."] [2024/05/30 15:01:44.518 +08:00] [INFO] [main.go:180] ["genLock started"] [2024/05/30 15:01:44.519 +08:00] [INFO] [region_cache.go:2683] ["change store resolve state"] [store=1] [addr=127.0.0.1:20161] [from=unresolved] [to=resolved] [liveness-state=reachable] cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:01:48 CST 2024] <<<<<< run test case resourcecontrol success! >>>>>> table ddl_only_block_related_table.finish_mark not exists for 4-th check, retry later table savepoint.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... table processor_delay.t48 exists table processor_delay.t49 not exists for 1-th check, retry later table ddl_only_block_related_table.finish_mark exists wait process 7836 exit for 1-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 15:01:50 CST 2024] <<<<<< run test case savepoint success! >>>>>> wait process 7836 exit for 2-th time... wait process 7836 exit for 3-th time... /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (7836) - No such process wait process 7836 exit for 4-th time... process 7836 already exit [Thu May 30 15:01:52 CST 2024] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteNotDone=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.79897991.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t49 exists table processor_delay.t50 not exists for 1-th check, retry later table processor_delay.t50 exists check diff failed 1-th time, retry later [2024/05/30 15:01:54.526 +08:00] [INFO] [main.go:196] ["genLock done"] [2024/05/30 15:01:54.526 +08:00] [INFO] [tso_dispatcher.go:186] ["exit tso requests cancel loop"] [2024/05/30 15:01:54.526 +08:00] [INFO] [tso_dispatcher.go:410] ["[tso] stop fetching the pending tso requests due to context canceled"] [dc-location=global] [2024/05/30 15:01:54.526 +08:00] [INFO] [tso_dispatcher.go:240] ["exit tso dispatcher loop"] [2024/05/30 15:01:54.526 +08:00] [INFO] [tso_dispatcher.go:344] ["[tso] exit tso dispatcher"] [dc-location=global] [2024/05/30 15:01:54.526 +08:00] [INFO] [pd_service_discovery.go:248] ["[pd] exit member loop due to context canceled"] [2024/05/30 15:01:54.526 +08:00] [INFO] [resource_manager_client.go:271] ["[resource manager] exit resource token dispatcher"] [2024/05/30 15:01:54.526 +08:00] [INFO] [tso_client.go:134] ["closing tso client"] [2024/05/30 15:01:54.526 +08:00] [INFO] [tso_client.go:139] ["close tso client"] [2024/05/30 15:01:54.526 +08:00] [INFO] [tso_client.go:150] ["tso client is closed"] [2024/05/30 15:01:54.526 +08:00] [INFO] [pd_service_discovery.go:294] ["[pd] close pd service discovery client"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:01:55 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7374684260771247141 Namespace:default ID:ddl-only-block-related-table SinkURI:pulsar://127.0.0.1:6650/ticdc-common-1-test-8611?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 15:01:40.057427955 +0800 CST StartTs:450115010530115586 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004736000 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115010569437187} {CheckpointTs:450115014291881985 MinTableBarrierTs:450115014291881985 AdminJobType:noop} span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 450115014291881985, checkpointTs: 450115014291881985, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115014291881988, checkpointTs: 450115014291881985, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450115014291881985, checkpointTs: 450115014291881985, state: Replicating span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 450115014291881985, checkpointTs: 450115014291881985, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9a0ab43f-08d4-452a-bb7b-c39b1e6567af {"id":"9a0ab43f-08d4-452a-bb7b-c39b1e6567af","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84da7687a 9a0ab43f-08d4-452a-bb7b-c39b1e6567af /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7374684260771247141,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"pulsar://127.0.0.1:6650/ticdc-common-1-test-8611?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T15:01:40.057427955+08:00","start-ts":450115010530115586,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115010569437187} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":450115014291881985,"min-table-barrier-ts":450115014291881985,"admin-job-type":0} /tidb/cdc/default/default/task/position/9a0ab43f-08d4-452a-bb7b-c39b1e6567af/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684260771247141 {"id":7374684260771247141,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7374684260771247141 Namespace:default ID:ddl-only-block-related-table SinkURI:pulsar://127.0.0.1:6650/ticdc-common-1-test-8611?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 15:01:40.057427955 +0800 CST StartTs:450115010530115586 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004736000 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115010569437187} {CheckpointTs:450115014291881985 MinTableBarrierTs:450115014291881985 AdminJobType:noop} span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 450115014291881985, checkpointTs: 450115014291881985, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115014291881988, checkpointTs: 450115014291881985, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450115014291881985, checkpointTs: 450115014291881985, state: Replicating span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 450115014291881985, checkpointTs: 450115014291881985, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9a0ab43f-08d4-452a-bb7b-c39b1e6567af {"id":"9a0ab43f-08d4-452a-bb7b-c39b1e6567af","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84da7687a 9a0ab43f-08d4-452a-bb7b-c39b1e6567af /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7374684260771247141,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"pulsar://127.0.0.1:6650/ticdc-common-1-test-8611?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T15:01:40.057427955+08:00","start-ts":450115010530115586,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115010569437187} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":450115014291881985,"min-table-barrier-ts":450115014291881985,"admin-job-type":0} /tidb/cdc/default/default/task/position/9a0ab43f-08d4-452a-bb7b-c39b1e6567af/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684260771247141 {"id":7374684260771247141,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7374684260771247141 Namespace:default ID:ddl-only-block-related-table SinkURI:pulsar://127.0.0.1:6650/ticdc-common-1-test-8611?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 15:01:40.057427955 +0800 CST StartTs:450115010530115586 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004736000 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115010569437187} {CheckpointTs:450115014291881985 MinTableBarrierTs:450115014291881985 AdminJobType:noop} span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 450115014291881985, checkpointTs: 450115014291881985, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115014291881988, checkpointTs: 450115014291881985, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450115014291881985, checkpointTs: 450115014291881985, state: Replicating span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 450115014291881985, checkpointTs: 450115014291881985, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9a0ab43f-08d4-452a-bb7b-c39b1e6567af {"id":"9a0ab43f-08d4-452a-bb7b-c39b1e6567af","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84da7687a 9a0ab43f-08d4-452a-bb7b-c39b1e6567af /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7374684260771247141,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"pulsar://127.0.0.1:6650/ticdc-common-1-test-8611?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T15:01:40.057427955+08:00","start-ts":450115010530115586,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115010569437187} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":450115014291881985,"min-table-barrier-ts":450115014291881985,"admin-job-type":0} /tidb/cdc/default/default/task/position/9a0ab43f-08d4-452a-bb7b-c39b1e6567af/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684260771247141 {"id":7374684260771247141,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_ts_not_forward ddl-only-block-related-table check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 15:01:58 CST 2024] <<<<<< run test case processor_etcd_worker_delay success! >>>>>> \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/run.sh using Sink-Type: pulsar... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status + source /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/synced_status + CDC_BINARY=cdc.test + SINK_TYPE=pulsar + CDC_COUNT=3 + DB_COUNT=4 + trap stop_tidb_cluster EXIT + run_normal_case_and_unavailable_pd conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status The 1 times to try to start tidb cluster... run task failed 1-th time, retry later table test.t2 not exists for 1-th check, retry later table test.t2 not exists for 2-th check, retry later check_ts_not_forward ddl-only-block-related-table start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... table test.t2 not exists for 3-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table test.t2 not exists for 4-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/sink_hang/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table test.t2 not exists for 5-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/sink_hang Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... table test.t2 not exists for 6-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.t2 not exists for 7-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2139a1d80008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:15338, start at 2024-05-30 15:02:12.863884746 +0800 CST m=+5.311685482 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:12.871 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:12.854 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:12.854 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.t2 exists check diff successfully Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release wait process cdc.test exit for 1-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2139a1d80008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:15338, start at 2024-05-30 15:02:12.863884746 +0800 CST m=+5.311685482 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:12.871 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:12.854 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:12.854 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2139a0fc0017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:15428, start at 2024-05-30 15:02:12.849276407 +0800 CST m=+5.243004249 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:12.858 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:12.849 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:12.849 +0800 All versions after safe point can be accessed. (DO NOT EDIT) wait process cdc.test exit for 2-th time... Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:02:16 CST 2024] <<<<<< run test case resolve_lock success! >>>>>> Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.16680.out cli tso query --pd=http://127.0.0.1:2379 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + tso='450115020588056577 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115020588056577 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=450115020588056577 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Thu May 30 15:02:19 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.1671416716.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:02:22 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2ab3512f-bf2b-4743-b53e-258a2291e4b8 {"id":"2ab3512f-bf2b-4743-b53e-258a2291e4b8","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84e41c6c2 2ab3512f-bf2b-4743-b53e-258a2291e4b8 /tidb/cdc/default/default/upstream/7374684436467211107 {"id":7374684436467211107,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2ab3512f-bf2b-4743-b53e-258a2291e4b8 {"id":"2ab3512f-bf2b-4743-b53e-258a2291e4b8","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84e41c6c2 2ab3512f-bf2b-4743-b53e-258a2291e4b8 /tidb/cdc/default/default/upstream/7374684436467211107 {"id":7374684436467211107,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2ab3512f-bf2b-4743-b53e-258a2291e4b8 {"id":"2ab3512f-bf2b-4743-b53e-258a2291e4b8","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84e41c6c2 2ab3512f-bf2b-4743-b53e-258a2291e4b8 /tidb/cdc/default/default/upstream/7374684436467211107 {"id":7374684436467211107,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=450115020588056577 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.16762.out cli changefeed create --start-ts=450115020588056577 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213a2f580018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-k5r94, pid:7629, start at 2024-05-30 15:02:21.950857672 +0800 CST m=+5.235442756 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:21.957 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:21.960 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:21.960 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Create changefeed successfully! ID: test-1 Info: {"upstream_id":7374684436467211107,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T15:02:23.219576552+08:00","start_ts":450115020588056577,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115020588056577,"checkpoint_ts":450115020588056577,"checkpoint_time":"2024-05-30 15:02:18.254"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 9013 0 --:--:-- --:--:-- --:--:-- 9208 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-30 15:02:24.604","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-05-30 15:02:24.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:02:24.604","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-30' '15:02:24.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:02:24.604","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-30' '15:02:24.000","info":"Data' syncing is 'finished"}' ++ jq -r .sink_checkpoint_ts + sink_checkpoint_ts='2024-05-30 15:02:24.604' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:02:24.604","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-30' '15:02:24.000","info":"Data' syncing is 'finished"}' ++ jq -r .puller_resolved_ts + puller_resolved_ts='1970-01-01 08:00:00.000' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:02:24.604","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-30' '15:02:24.000","info":"Data' syncing is 'finished"}' ++ jq -r .last_synced_ts + last_synced_ts='1970-01-01 08:00:00.000' + '[' true '!=' true ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' ++ date '+%Y-%m-%d %H:%M:%S' + current='2024-05-30 15:02:24' + echo 'sink_checkpoint_ts is 2024-05-30' 15:02:24.604 sink_checkpoint_ts is 2024-05-30 15:02:24.604 ++ date -d '2024-05-30 15:02:24.604' +%s + checkpoint_timestamp=1717052544 ++ date -d '2024-05-30 15:02:24' +%s + current_timestamp=1717052544 + '[' 0 -gt 300 ']' + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213a2f580018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-k5r94, pid:7629, start at 2024-05-30 15:02:21.950857672 +0800 CST m=+5.235442756 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:21.957 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:21.960 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:21.960 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213a31100015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-k5r94, pid:7711, start at 2024-05-30 15:02:22.045522701 +0800 CST m=+5.264028938 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:22.056 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:22.020 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:22.020 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sink_hang/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sink_hang/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/move_table/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... [Thu May 30 15:02:27 CST 2024] <<<<<< START cdc server in sink_hang case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkExecDMLError=2*return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_hang.89558957.out server --log-file /tmp/tidb_cdc_test/sink_hang/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sink_hang/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table test.t1 exists + sleep 5 start tidb cluster in /tmp/tidb_cdc_test/move_table Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:02:30 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7036396c-5986-4810-b954-16c1567fafae {"id":"7036396c-5986-4810-b954-16c1567fafae","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84e5fa6bd 7036396c-5986-4810-b954-16c1567fafae /tidb/cdc/default/default/upstream/7374684476921666744 {"id":7374684476921666744,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7036396c-5986-4810-b954-16c1567fafae {"id":"7036396c-5986-4810-b954-16c1567fafae","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84e5fa6bd 7036396c-5986-4810-b954-16c1567fafae /tidb/cdc/default/default/upstream/7374684476921666744 {"id":7374684476921666744,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7036396c-5986-4810-b954-16c1567fafae {"id":"7036396c-5986-4810-b954-16c1567fafae","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84e5fa6bd 7036396c-5986-4810-b954-16c1567fafae /tidb/cdc/default/default/upstream/7374684476921666744 {"id":7374684476921666744,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Thu May 30 15:02:30 CST 2024] <<<<<< START Pulsar consumer in sink_hang case >>>>>> check_changefeed_status 127.0.0.1:8300 d444e63b-0966-4c86-9883-6e40c256bb34 normal last_error null + endpoint=127.0.0.1:8300 + changefeed_id=d444e63b-0966-4c86-9883-6e40c256bb34 + expected_state=normal + field=last_error + error_pattern=null ++ curl 127.0.0.1:8300/api/v2/changefeeds/d444e63b-0966-4c86-9883-6e40c256bb34/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 86 100 86 0 0 736 0 --:--:-- --:--:-- --:--:-- 741 + info='{"state":"normal","resolved_ts":450115023722512390,"checkpoint_ts":450115023722512390}' + echo '{"state":"normal","resolved_ts":450115023722512390,"checkpoint_ts":450115023722512390}' {"state":"normal","resolved_ts":450115023722512390,"checkpoint_ts":450115023722512390} ++ echo '{"state":"normal","resolved_ts":450115023722512390,"checkpoint_ts":450115023722512390}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] + [[ -z last_error ]] ++ echo '{"state":"normal","resolved_ts":450115023722512390,"checkpoint_ts":450115023722512390}' ++ jq -r .last_error.message + error_msg=null + [[ ! null =~ null ]] run task successfully check_changefeed_status 127.0.0.1:8300 d444e63b-0966-4c86-9883-6e40c256bb34 normal last_warning null + endpoint=127.0.0.1:8300 + changefeed_id=d444e63b-0966-4c86-9883-6e40c256bb34 + expected_state=normal + field=last_warning + error_pattern=null ++ curl 127.0.0.1:8300/api/v2/changefeeds/d444e63b-0966-4c86-9883-6e40c256bb34/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 86 100 86 0 0 658 0 --:--:-- --:--:-- --:--:-- 661 + info='{"state":"normal","resolved_ts":450115023722512390,"checkpoint_ts":450115023722512390}' + echo '{"state":"normal","resolved_ts":450115023722512390,"checkpoint_ts":450115023722512390}' {"state":"normal","resolved_ts":450115023722512390,"checkpoint_ts":450115023722512390} ++ echo '{"state":"normal","resolved_ts":450115023722512390,"checkpoint_ts":450115023722512390}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] + [[ -z last_warning ]] ++ echo '{"state":"normal","resolved_ts":450115023722512390,"checkpoint_ts":450115023722512390}' ++ jq -r .last_warning.message + error_msg=null + [[ ! null =~ null ]] run task successfully table sink_hang.t1 not exists for 1-th check, retry later ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 1965 0 --:--:-- --:--:-- --:--:-- 1975 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-30 15:02:31.604","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"2024-05-30 15:02:24.904","now_ts":"2024-05-30 15:02:32.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:02:31.604","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-05-30' '15:02:24.904","now_ts":"2024-05-30' '15:02:32.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:02:31.604","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-05-30' '15:02:24.904","now_ts":"2024-05-30' '15:02:32.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 table sink_hang.t1 exists table sink_hang.t2 not exists for 1-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table sink_hang.t2 exists check diff failed 1-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 3-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213b4ee40007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-179cx, pid:21632, start at 2024-05-30 15:02:40.319980426 +0800 CST m=+5.641779993 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:40.327 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:40.313 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:40.313 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 4-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213b4ee40007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-179cx, pid:21632, start at 2024-05-30 15:02:40.319980426 +0800 CST m=+5.641779993 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:40.327 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:40.313 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:40.313 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213b508c0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-179cx, pid:21709, start at 2024-05-30 15:02:40.462983007 +0800 CST m=+5.688224276 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:04:40.473 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:02:40.469 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:52:40.469 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/move_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/move_table/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/move_table/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/move_table/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/move_table/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 5-th time, retry later + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.cli.23020.out cli tso query --pd=http://127.0.0.1:2379 check diff failed 6-th time, retry later + set +x + tso='450115027707625473 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115027707625473 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ***************** properties ***************** "requestdistribution"="uniform" "mysql.port"="4000" "scanproportion"="0" "mysql.user"="root" "operationcount"="0" "readproportion"="0" "threadcount"="10" "updateproportion"="0" "dotransactions"="false" "mysql.host"="127.0.0.1" "insertproportion"="0" "workload"="core" "recordcount"="10000" "mysql.db"="move_table" "readallfields"="true" ********************************************** check diff failed 7-th time, retry later check diff failed 8-th time, retry later Run finished, takes 4.306427597s INSERT - Takes(s): 4.3, Count: 10000, OPS: 2325.6, Avg(us): 4242, Min(us): 806, Max(us): 89199, 95th(us): 7000, 99th(us): 81000 [Thu May 30 15:02:51 CST 2024] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.2310623108.out server --log-file /tmp/tidb_cdc_test/move_table/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data1 --cluster-id default --addr 127.0.0.1:8300 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 9-th time, retry later check diff successfully wait process cdc.test exit for 1-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:02:54 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6f77f2cc-40d7-4c08-95fc-e9e78b98b116 {"id":"6f77f2cc-40d7-4c08-95fc-e9e78b98b116","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ea9b4e6 6f77f2cc-40d7-4c08-95fc-e9e78b98b116 /tidb/cdc/default/default/upstream/7374684556279632194 {"id":7374684556279632194,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6f77f2cc-40d7-4c08-95fc-e9e78b98b116 {"id":"6f77f2cc-40d7-4c08-95fc-e9e78b98b116","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ea9b4e6 6f77f2cc-40d7-4c08-95fc-e9e78b98b116 /tidb/cdc/default/default/upstream/7374684556279632194 {"id":7374684556279632194,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6f77f2cc-40d7-4c08-95fc-e9e78b98b116 {"id":"6f77f2cc-40d7-4c08-95fc-e9e78b98b116","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ea9b4e6 6f77f2cc-40d7-4c08-95fc-e9e78b98b116 /tidb/cdc/default/default/upstream/7374684556279632194 {"id":7374684556279632194,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.cli.23150.out cli changefeed create --start-ts=450115027707625473 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-move-table-test-29200?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: e32af784-e449-4e4a-aa5d-f9dbc2f3a33f Info: {"upstream_id":7374684556279632194,"namespace":"default","id":"e32af784-e449-4e4a-aa5d-f9dbc2f3a33f","sink_uri":"pulsar://127.0.0.1:6650/ticdc-move-table-test-29200?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T15:02:55.041702819+08:00","start_ts":450115027707625473,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115027707625473,"checkpoint_ts":450115027707625473,"checkpoint_time":"2024-05-30 15:02:45.413"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 15:02:55 CST 2024] <<<<<< run test case sink_hang success! >>>>>> + set +x [Thu May 30 15:02:56 CST 2024] <<<<<< START Pulsar consumer in move_table case >>>>>> [Thu May 30 15:02:56 CST 2024] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.2319723199.out server --log-file /tmp/tidb_cdc_test/move_table/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data2 --cluster-id default --addr 127.0.0.1:8301 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:02:59 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {UpstreamID:7374684556279632194 Namespace:default ID:e32af784-e449-4e4a-aa5d-f9dbc2f3a33f SinkURI:pulsar://127.0.0.1:6650/ticdc-move-table-test-29200?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 15:02:55.041702819 +0800 CST StartTs:450115027707625473 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004010480 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115030224207876} {CheckpointTs:450115028127318038 MinTableBarrierTs:450115031207247879 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6f77f2cc-40d7-4c08-95fc-e9e78b98b116 {"id":"6f77f2cc-40d7-4c08-95fc-e9e78b98b116","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8 {"id":"cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ea9b4e6 6f77f2cc-40d7-4c08-95fc-e9e78b98b116 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ea9b536 cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8 /tidb/cdc/default/default/changefeed/info/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"upstream-id":7374684556279632194,"namespace":"default","changefeed-id":"e32af784-e449-4e4a-aa5d-f9dbc2f3a33f","sink-uri":"pulsar://127.0.0.1:6650/ticdc-move-table-test-29200?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T15:02:55.041702819+08:00","start-ts":450115027707625473,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115030224207876} /tidb/cdc/default/default/changefeed/status/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"checkpoint-ts":450115028127318038,"min-table-barrier-ts":450115031207247879,"admin-job-type":0} /tidb/cdc/default/default/task/position/6f77f2cc-40d7-4c08-95fc-e9e78b98b116/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684556279632194 {"id":7374684556279632194,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {UpstreamID:7374684556279632194 Namespace:default ID:e32af784-e449-4e4a-aa5d-f9dbc2f3a33f SinkURI:pulsar://127.0.0.1:6650/ticdc-move-table-test-29200?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 15:02:55.041702819 +0800 CST StartTs:450115027707625473 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004010480 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115030224207876} {CheckpointTs:450115028127318038 MinTableBarrierTs:450115031207247879 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6f77f2cc-40d7-4c08-95fc-e9e78b98b116 {"id":"6f77f2cc-40d7-4c08-95fc-e9e78b98b116","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8 {"id":"cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ea9b4e6 6f77f2cc-40d7-4c08-95fc-e9e78b98b116 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ea9b536 cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8 /tidb/cdc/default/default/changefeed/info/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"upstream-id":7374684556279632194,"namespace":"default","changefeed-id":"e32af784-e449-4e4a-aa5d-f9dbc2f3a33f","sink-uri":"pulsar://127.0.0.1:6650/ticdc-move-table-test-29200?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T15:02:55.041702819+08:00","start-ts":450115027707625473,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115030224207876} /tidb/cdc/default/default/changefeed/status/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"checkpoint-ts":450115028127318038,"min-table-barrier-ts":450115031207247879,"admin-job-type":0} /tidb/cdc/default/default/task/position/6f77f2cc-40d7-4c08-95fc-e9e78b98b116/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684556279632194 {"id":7374684556279632194,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {UpstreamID:7374684556279632194 Namespace:default ID:e32af784-e449-4e4a-aa5d-f9dbc2f3a33f SinkURI:pulsar://127.0.0.1:6650/ticdc-move-table-test-29200?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 15:02:55.041702819 +0800 CST StartTs:450115027707625473 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004010480 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115030224207876} {CheckpointTs:450115028127318038 MinTableBarrierTs:450115031207247879 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6f77f2cc-40d7-4c08-95fc-e9e78b98b116 {"id":"6f77f2cc-40d7-4c08-95fc-e9e78b98b116","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8 {"id":"cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ea9b4e6 6f77f2cc-40d7-4c08-95fc-e9e78b98b116 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ea9b536 cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8 /tidb/cdc/default/default/changefeed/info/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"upstream-id":7374684556279632194,"namespace":"default","changefeed-id":"e32af784-e449-4e4a-aa5d-f9dbc2f3a33f","sink-uri":"pulsar://127.0.0.1:6650/ticdc-move-table-test-29200?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T15:02:55.041702819+08:00","start-ts":450115027707625473,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115030224207876} /tidb/cdc/default/default/changefeed/status/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"checkpoint-ts":450115028127318038,"min-table-barrier-ts":450115031207247879,"admin-job-type":0} /tidb/cdc/default/default/task/position/6f77f2cc-40d7-4c08-95fc-e9e78b98b116/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684556279632194 {"id":7374684556279632194,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Thu May 30 15:02:59 CST 2024] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.2331423316.out server --log-file /tmp/tidb_cdc_test/move_table/cdc3.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data3 --cluster-id default --addr 127.0.0.1:8302 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:03:02 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {UpstreamID:7374684556279632194 Namespace:default ID:e32af784-e449-4e4a-aa5d-f9dbc2f3a33f SinkURI:pulsar://127.0.0.1:6650/ticdc-move-table-test-29200?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 15:02:55.041702819 +0800 CST StartTs:450115027707625473 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003666c60 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115030224207876} {CheckpointTs:450115028166377477 MinTableBarrierTs:450115032282300419 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1bce962d-74d3-40b8-9faa-f543f8937971 {"id":"1bce962d-74d3-40b8-9faa-f543f8937971","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/6f77f2cc-40d7-4c08-95fc-e9e78b98b116 {"id":"6f77f2cc-40d7-4c08-95fc-e9e78b98b116","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8 {"id":"cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ea9b4e6 6f77f2cc-40d7-4c08-95fc-e9e78b98b116 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ea9b536 cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ea9b567 1bce962d-74d3-40b8-9faa-f543f8937971 /tidb/cdc/default/default/changefeed/info/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"upstream-id":7374684556279632194,"namespace":"default","changefeed-id":"e32af784-e449-4e4a-aa5d-f9dbc2f3a33f","sink-uri":"pulsar://127.0.0.1:6650/ticdc-move-table-test-29200?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T15:02:55.041702819+08:00","start-ts":450115027707625473,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115030224207876} /tidb/cdc/default/default/changefeed/status/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"checkpoint-ts":450115028166377477,"min-table-barrier-ts":450115032282300419,"admin-job-type":0} /tidb/cdc/default/default/task/position/1bce962d-74d3-40b8-9faa-f543f8937971/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/6f77f2cc-40d7-4c08-95fc-e9e78b98b116/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684556279632194 {"id":7374684556279632194,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {UpstreamID:7374684556279632194 Namespace:default ID:e32af784-e449-4e4a-aa5d-f9dbc2f3a33f SinkURI:pulsar://127.0.0.1:6650/ticdc-move-table-test-29200?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 15:02:55.041702819 +0800 CST StartTs:450115027707625473 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003666c60 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115030224207876} {CheckpointTs:450115028166377477 MinTableBarrierTs:450115032282300419 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1bce962d-74d3-40b8-9faa-f543f8937971 {"id":"1bce962d-74d3-40b8-9faa-f543f8937971","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/6f77f2cc-40d7-4c08-95fc-e9e78b98b116 {"id":"6f77f2cc-40d7-4c08-95fc-e9e78b98b116","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8 {"id":"cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ea9b4e6 6f77f2cc-40d7-4c08-95fc-e9e78b98b116 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ea9b536 cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ea9b567 1bce962d-74d3-40b8-9faa-f543f8937971 /tidb/cdc/default/default/changefeed/info/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"upstream-id":7374684556279632194,"namespace":"default","changefeed-id":"e32af784-e449-4e4a-aa5d-f9dbc2f3a33f","sink-uri":"pulsar://127.0.0.1:6650/ticdc-move-table-test-29200?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T15:02:55.041702819+08:00","start-ts":450115027707625473,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115030224207876} /tidb/cdc/default/default/changefeed/status/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"checkpoint-ts":450115028166377477,"min-table-barrier-ts":450115032282300419,"admin-job-type":0} /tidb/cdc/default/default/task/position/1bce962d-74d3-40b8-9faa-f543f8937971/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/6f77f2cc-40d7-4c08-95fc-e9e78b98b116/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684556279632194 {"id":7374684556279632194,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {UpstreamID:7374684556279632194 Namespace:default ID:e32af784-e449-4e4a-aa5d-f9dbc2f3a33f SinkURI:pulsar://127.0.0.1:6650/ticdc-move-table-test-29200?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 15:02:55.041702819 +0800 CST StartTs:450115027707625473 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003666c60 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115030224207876} {CheckpointTs:450115028166377477 MinTableBarrierTs:450115032282300419 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1bce962d-74d3-40b8-9faa-f543f8937971 {"id":"1bce962d-74d3-40b8-9faa-f543f8937971","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/6f77f2cc-40d7-4c08-95fc-e9e78b98b116 {"id":"6f77f2cc-40d7-4c08-95fc-e9e78b98b116","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8 {"id":"cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ea9b4e6 6f77f2cc-40d7-4c08-95fc-e9e78b98b116 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ea9b536 cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8 /tidb/cdc/default/__cdc_meta__/owner/22318fc84ea9b567 1bce962d-74d3-40b8-9faa-f543f8937971 /tidb/cdc/default/default/changefeed/info/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"upstream-id":7374684556279632194,"namespace":"default","changefeed-id":"e32af784-e449-4e4a-aa5d-f9dbc2f3a33f","sink-uri":"pulsar://127.0.0.1:6650/ticdc-move-table-test-29200?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T15:02:55.041702819+08:00","start-ts":450115027707625473,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115030224207876} /tidb/cdc/default/default/changefeed/status/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"checkpoint-ts":450115028166377477,"min-table-barrier-ts":450115032282300419,"admin-job-type":0} /tidb/cdc/default/default/task/pos+ grep -q 'etcd info' ition/1bce962d-74d3-40b8-9faa-f543f8937971/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/6f77f2cc-40d7-4c08-95fc-e9e78b98b116/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8/e32af784-e449-4e4a-aa5d-f9dbc2f3a33f {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684556279632194 {"id":7374684556279632194,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x table move_table.usertable exists go: downloading github.com/benbjohnson/clock v1.3.5 go: downloading github.com/IBM/sarama v1.41.2 go: downloading go.etcd.io/etcd/server/v3 v3.5.12 go: downloading github.com/xdg/scram v1.0.5 go: downloading github.com/go-mysql-org/go-mysql v1.7.1-0.20240507075657-2bd4573edde2 go: downloading github.com/apache/pulsar-client-go v0.11.0 go: downloading github.com/aws/aws-sdk-go-v2 v1.19.1 go: downloading github.com/cakturk/go-netstat v0.0.0-20200220111822-e5b49efee7a5 go: downloading github.com/KimMachineGun/automemlimit v0.2.4 go: downloading github.com/tinylib/msgp v1.1.6 go: downloading github.com/xdg/stringprep v1.0.3 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/containerd/cgroups v1.0.4 go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2 go: downloading github.com/stretchr/objx v0.5.2 go: downloading github.com/philhofer/fwd v1.1.1 go: downloading github.com/opencontainers/runtime-spec v1.0.2 go: downloading github.com/godbus/dbus/v5 v5.0.4 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading github.com/cilium/ebpf v0.4.0 go: downloading github.com/siddontang/go v0.0.0-20180604090527-bdc77568d726 go: downloading github.com/siddontang/go-log v0.0.0-20180807004314-8d05993dda07 go: downloading github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 go: downloading github.com/aws/smithy-go v1.13.5 go: downloading github.com/eapache/go-resiliency v1.4.0 go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 go: downloading github.com/jcmturner/gokrb5/v8 v8.4.4 go: downloading github.com/eapache/queue v1.1.0 go: downloading github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 go: downloading github.com/jcmturner/gofork v1.7.6 go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/pierrec/lz4 v2.6.1+incompatible go: downloading github.com/spaolacci/murmur3 v1.1.0 go: downloading github.com/99designs/keyring v1.2.1 go: downloading github.com/bits-and-blooms/bitset v1.4.0 go: downloading github.com/linkedin/goavro/v2 v2.11.1 go: downloading golang.org/x/mod v0.17.0 go: downloading github.com/AthenZ/athenz v1.10.39 go: downloading github.com/hashicorp/errwrap v1.0.0 go: downloading github.com/mtibben/percent v0.2.1 go: downloading github.com/dvsekhvalnov/jose2go v1.5.0 go: downloading github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c go: downloading go.etcd.io/etcd/pkg/v3 v3.5.12 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading go.etcd.io/bbolt v1.3.9 go: downloading github.com/soheilhy/cmux v0.1.5 go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0 go: downloading github.com/jonboulle/clockwork v0.4.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.22.0 go: downloading go.opentelemetry.io/otel/sdk v1.22.0 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 go: downloading go.etcd.io/etcd/raft/v3 v3.5.12 go: downloading github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 go: downloading go.etcd.io/etcd/client/v2 v2.305.12 go: downloading github.com/golang-jwt/jwt/v4 v4.5.0 go: downloading github.com/hashicorp/go-uuid v1.0.3 go: downloading github.com/jcmturner/dnsutils/v2 v2.0.0 go: downloading github.com/jcmturner/aescts/v2 v2.0.0 go: downloading github.com/jcmturner/rpc/v2 v2.0.3 go: downloading github.com/gorilla/websocket v1.5.1 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 go: downloading go.opentelemetry.io/proto/otlp v1.1.0 go: downloading github.com/cenkalti/backoff/v4 v4.2.1 go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } go: downloading github.com/ardielle/ardielle-go v1.5.2 [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } pass check, checkpoint tso not forward after 10s run task successfully wait process 7994 exit for 1-th time... wait process 7994 exit for 2-th time... wait process 7994 exit for 3-th time... wait process 7994 exit for 4-th time... /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (7994) - No such process wait process 7994 exit for 5-th time... process 7994 already exit [Thu May 30 15:03:05 CST 2024] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.85678569.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:03:08 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7374684260771247141 Namespace:default ID:ddl-only-block-related-table SinkURI:pulsar://127.0.0.1:6650/ticdc-common-1-test-8611?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 15:01:40.057427955 +0800 CST StartTs:450115010530115586 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0040145a0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115010569437187} {CheckpointTs:450115014580240394 MinTableBarrierTs:450115033441763331 AdminJobType:noop} span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 450115014580240394, checkpointTs: 450115014580240394, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115014580240394, checkpointTs: 450115014580240394, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450115014580240394, checkpointTs: 450115014580240394, state: Preparing span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 450115014580240394, checkpointTs: 450115014580240394, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/097f25dd-84ea-4f4a-bb17-2fa9dee09f79 {"id":"097f25dd-84ea-4f4a-bb17-2fa9dee09f79","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84da76a43 097f25dd-84ea-4f4a-bb17-2fa9dee09f79 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7374684260771247141,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"pulsar://127.0.0.1:6650/ticdc-common-1-test-8611?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T15:01:40.057427955+08:00","start-ts":450115010530115586,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115010569437187} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":450115014580240394,"min-table-barrier-ts":450115033441763331,"admin-job-type":0} /tidb/cdc/default/default/task/position/097f25dd-84ea-4f4a-bb17-2fa9dee09f79/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684260771247141 {"id":7374684260771247141,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7374684260771247141 Namespace:default ID:ddl-only-block-related-table SinkURI:pulsar://127.0.0.1:6650/ticdc-common-1-test-8611?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 15:01:40.057427955 +0800 CST StartTs:450115010530115586 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0040145a0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115010569437187} {CheckpointTs:450115014580240394 MinTableBarrierTs:450115033441763331 AdminJobType:noop} span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 450115014580240394, checkpointTs: 450115014580240394, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115014580240394, checkpointTs: 450115014580240394, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450115014580240394, checkpointTs: 450115014580240394, state: Preparing span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 450115014580240394, checkpointTs: 450115014580240394, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/097f25dd-84ea-4f4a-bb17-2fa9dee09f79 {"id":"097f25dd-84ea-4f4a-bb17-2fa9dee09f79","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84da76a43 097f25dd-84ea-4f4a-bb17-2fa9dee09f79 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7374684260771247141,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"pulsar://127.0.0.1:6650/ticdc-common-1-test-8611?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T15:01:40.057427955+08:00","start-ts":450115010530115586,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":450115010569437187} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":450115014580240394,"min-table-barrier-ts":450115033441763331,"admin-job-type":0} /tidb/cdc/default/default/task/position/097f25dd-84ea-4f4a-bb17-2fa9dee09f79/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684260771247141 {"id":7374684260771247141,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7374684260771247141 Namespace:default ID:ddl-only-block-related-table SinkURI:pulsar://127.0.0.1:6650/ticdc-common-1-test-8611?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-30 15:01:40.057427955 +0800 CST StartTs:450115010530115586 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0040145a0 State:normal Error: Warning: CreatorVersion:v7.5.1-58-g8f92142d7 Epoch:450115010569437187} {CheckpointTs:450115014580240394 MinTableBarrierTs:450115033441763331 AdminJobType:noop} span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 450115014580240394, checkpointTs: 450115014580240394, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 450115014580240394, checkpointTs: 450115014580240394, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 450115014580240394, checkpointTs: 450115014580240394, state: Preparing span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 450115014580240394, checkpointTs: 450115014580240394, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/097f25dd-84ea-4f4a-bb17-2fa9dee09f79 {"id":"097f25dd-84ea-4f4a-bb17-2fa9dee09f79","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84da76a43 097f25dd-84ea-4f4a-bb17-2fa9dee09f79 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7374684260771247141,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"pulsar://127.0.0.1:6650/ticdc-common-1-test-8611?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-30T15:01:40.057427955+08:00","start-ts":450115010530115586,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"open":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-58-g8f92142d7","epoch":45011501056943718+ grep -q 'etcd info' 7} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":450115014580240394,"min-table-barrier-ts":450115033441763331,"admin-job-type":0} /tidb/cdc/default/default/task/position/097f25dd-84ea-4f4a-bb17-2fa9dee09f79/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7374684260771247141 {"id":7374684260771247141,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x check diff failed 1-th time, retry later check diff successfully check_ts_forward ddl-only-block-related-table changefeed is working normally rts: 450115034228195300->450115034490339300 checkpoint: 450115034228195300->450115034490339300 run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 15:03:14 CST 2024] <<<<<< run test case ddl_only_block_related_table success! >>>>>> table big_txn.finish_mark_1 exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... wait process cdc.test exit for 4-th time... cdc.test: no process found wait process cdc.test exit for 5-th time... process cdc.test already exit [Thu May 30 15:03:15 CST 2024] <<<<<< run test case big_txn success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/changefeed_finish/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/changefeed_auto_stop/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... [2024/05/30 15:03:26.999 +08:00] [INFO] [main.go:61] ["table mover started"] [2024/05/30 15:03:27.001 +08:00] [INFO] [main.go:166] ["new cluster initialized"] [2024/05/30 15:03:27.002 +08:00] [DEBUG] [main.go:192] ["retrieved owner ID"] [ownerID=6f77f2cc-40d7-4c08-95fc-e9e78b98b116] [2024/05/30 15:03:27.002 +08:00] [DEBUG] [main.go:199] ["retrieved owner addr"] [ownerAddr=127.0.0.1:8300] [2024/05/30 15:03:27.003 +08:00] [DEBUG] [main.go:210] ["retrieved changefeeds"] [changefeedsError="json: unsupported type: map[model.ChangeFeedID]*mvccpb.KeyValue"] [2024/05/30 15:03:27.178 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=e32af784-e449-4e4a-aa5d-f9dbc2f3a33f] [captureID=1bce962d-74d3-40b8-9faa-f543f8937971] [processorDetail="{\"table_ids\":[]}"] [2024/05/30 15:03:27.377 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=e32af784-e449-4e4a-aa5d-f9dbc2f3a33f] [captureID=6f77f2cc-40d7-4c08-95fc-e9e78b98b116] [processorDetail="{\"table_ids\":[104]}"] [2024/05/30 15:03:27.577 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=e32af784-e449-4e4a-aa5d-f9dbc2f3a33f] [captureID=cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8] [processorDetail="{\"table_ids\":[]}"] [2024/05/30 15:03:27.577 +08:00] [INFO] [main.go:75] ["task status"] [status="{\"1bce962d-74d3-40b8-9faa-f543f8937971\":[],\"6f77f2cc-40d7-4c08-95fc-e9e78b98b116\":[{\"ID\":104,\"Changefeed\":\"e32af784-e449-4e4a-aa5d-f9dbc2f3a33f\"}],\"cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8\":[]}"] [2024/05/30 15:03:27.578 +08:00] [DEBUG] [main.go:288] ["preparing HTTP API call to owner"] [formStr="cf-id=e32af784-e449-4e4a-aa5d-f9dbc2f3a33f&target-cp-id=cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8&table-id=104"] [2024/05/30 15:03:27.665 +08:00] [INFO] [main.go:180] ["moved table successful"] [tableID=104] [2024/05/30 15:03:27.665 +08:00] [INFO] [main.go:114] ["all tables are moved"] [sourceCapture=6f77f2cc-40d7-4c08-95fc-e9e78b98b116] [targetCapture=cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8] table move_table.check1 not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/changefeed_finish Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/changefeed_auto_stop Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... table move_table.check1 not exists for 2-th check, retry later table move_table.check1 not exists for 3-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table move_table.check1 not exists for 4-th check, retry later Starting Upstream TiDB... Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table move_table.check1 not exists for 5-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table move_table.check1 not exists for 6-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table move_table.check1 not exists for 7-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213ef1e80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-mwr1g, pid:9710, start at 2024-05-30 15:03:39.922226748 +0800 CST m=+5.282559829 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:05:39.930 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:03:39.898 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:53:39.898 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table move_table.check1 not exists for 8-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213ef1e80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-mwr1g, pid:9710, start at 2024-05-30 15:03:39.922226748 +0800 CST m=+5.282559829 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:05:39.930 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:03:39.898 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:53:39.898 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213ef2940014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-mwr1g, pid:9799, start at 2024-05-30 15:03:39.969840353 +0800 CST m=+5.251295254 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:05:39.976 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:03:39.941 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:53:39.941 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213f069c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-0gwt8, pid:3986, start at 2024-05-30 15:03:41.239824303 +0800 CST m=+6.408219287 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:05:41.248 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:03:41.223 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:53:41.223 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213f069c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-0gwt8, pid:3986, start at 2024-05-30 15:03:41.239824303 +0800 CST m=+6.408219287 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:05:41.248 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:03:41.223 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:53:41.223 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f213f07940014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-0gwt8, pid:4061, start at 2024-05-30 15:03:41.325036509 +0800 CST m=+6.429840602 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:05:41.335 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:03:41.335 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:53:41.335 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_finish/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_finish/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table move_table.check1 not exists for 9-th check, retry later ***************** properties ***************** "insertproportion"="0" "readallfields"="true" "mysql.db"="changefeed_auto_stop_1" "recordcount"="20" "mysql.host"="127.0.0.1" "requestdistribution"="uniform" "workload"="core" "threadcount"="4" "scanproportion"="0" "readproportion"="0" "updateproportion"="0" "mysql.user"="root" "dotransactions"="false" "operationcount"="0" "mysql.port"="4000" ********************************************** Run finished, takes 10.768819ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3194.0, Avg(us): 1962, Min(us): 992, Max(us): 5158, 95th(us): 6000, 99th(us): 6000 ***************** properties ***************** "scanproportion"="0" "mysql.db"="changefeed_auto_stop_2" "operationcount"="0" "mysql.port"="4000" "workload"="core" "mysql.host"="127.0.0.1" "threadcount"="4" "dotransactions"="false" "requestdistribution"="uniform" "readproportion"="0" "mysql.user"="root" "updateproportion"="0" "recordcount"="20" "insertproportion"="0" "readallfields"="true" ********************************************** Run finished, takes 10.573908ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3475.6, Avg(us): 1971, Min(us): 1040, Max(us): 5066, 95th(us): 5000, 99th(us): 6000 ***************** properties ***************** "workload"="core" "updateproportion"="0" "insertproportion"="0" "scanproportion"="0" "threadcount"="4" "readallfields"="true" "mysql.host"="127.0.0.1" "mysql.port"="4000" "mysql.user"="root" "mysql.db"="changefeed_auto_stop_3" "operationcount"="0" "requestdistribution"="uniform" "recordcount"="20" "readproportion"="0" "dotransactions"="false" ********************************************** Run finished, takes 10.12898ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3367.1, Avg(us): 1902, Min(us): 1163, Max(us): 4116, 95th(us): 5000, 99th(us): 5000 [Thu May 30 15:03:45 CST 2024] <<<<<< START cdc server in changefeed_finish case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_finish.54255427.out server --log-file /tmp/tidb_cdc_test/changefeed_finish/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_finish/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ***************** properties ***************** "recordcount"="20" "dotransactions"="false" "mysql.db"="changefeed_auto_stop_4" "requestdistribution"="uniform" "operationcount"="0" "scanproportion"="0" "threadcount"="4" "workload"="core" "mysql.port"="4000" "mysql.user"="root" "readproportion"="0" "insertproportion"="0" "updateproportion"="0" "readallfields"="true" "mysql.host"="127.0.0.1" ********************************************** Run finished, takes 10.173148ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3582.4, Avg(us): 1888, Min(us): 993, Max(us): 4561, 95th(us): 5000, 99th(us): 5000 [Thu May 30 15:03:45 CST 2024] <<<<<< START cdc server in changefeed_auto_stop case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_auto_stop.1123911241.out server --log-file /tmp/tidb_cdc_test/changefeed_auto_stop/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_auto_stop/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table move_table.check1 not exists for 10-th check, retry later table move_table.check1 not exists for 11-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:03:48 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c0ea39c5-6ea6-4792-a2f1-af6911fceb00 {"id":"c0ea39c5-6ea6-4792-a2f1-af6911fceb00","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f8e74be c0ea39c5-6ea6-4792-a2f1-af6911fceb00 /tidb/cdc/default/default/upstream/7374684812134114084 {"id":7374684812134114084,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c0ea39c5-6ea6-4792-a2f1-af6911fceb00 {"id":"c0ea39c5-6ea6-4792-a2f1-af6911fceb00","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f8e74be c0ea39c5-6ea6-4792-a2f1-af6911fceb00 /tidb/cdc/default/default/upstream/7374684812134114084 {"id":7374684812134114084,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c0ea39c5-6ea6-4792-a2f1-af6911fceb00 {"id":"c0ea39c5-6ea6-4792-a2f1-af6911fceb00","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f8e74be c0ea39c5-6ea6-4792-a2f1-af6911fceb00 /tidb/cdc/default/default/upstream/7374684812134114084 {"id":7374684812134114084,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Thu May 30 15:03:48 CST 2024] <<<<<< START Pulsar consumer in changefeed_finish case >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > check diff failed 1-th time, retry later < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:03:49 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2aa4bf78-546d-4846-89d3-20642e8e49b2 {"id":"2aa4bf78-546d-4846-89d3-20642e8e49b2","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f901f2a 2aa4bf78-546d-4846-89d3-20642e8e49b2 /tidb/cdc/default/default/upstream/7374684810041326586 {"id":7374684810041326586,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2aa4bf78-546d-4846-89d3-20642e8e49b2 {"id":"2aa4bf78-546d-4846-89d3-20642e8e49b2","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f901f2a 2aa4bf78-546d-4846-89d3-20642e8e49b2 /tidb/cdc/default/default/upstream/7374684810041326586 {"id":7374684810041326586,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2aa4bf78-546d-4846-89d3-20642e8e49b2 {"id":"2aa4bf78-546d-4846-89d3-20642e8e49b2","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f901f2a 2aa4bf78-546d-4846-89d3-20642e8e49b2 /tidb/cdc/default/default/upstream/7374684810041326586 {"id":7374684810041326586,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Thu May 30 15:03:49 CST 2024] <<<<<< START cdc server in changefeed_auto_stop case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/pipeline/ProcessorSyncResolvedError=1*return(true);github.com/pingcap/tiflow/cdc/processor/ProcessorUpdatePositionDelaying=sleep(1000)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_auto_stop.1128311285.out server --log-file /tmp/tidb_cdc_test/changefeed_auto_stop/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_auto_stop/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table move_table.check1 not exists for 12-th check, retry later check diff failed 2-th time, retry later table move_table.check1 exists check diff successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:03:52 GMT < Content-Length: 867 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0a732fa0-3d34-4d9c-ad37-3369e6077cd0 {"id":"0a732fa0-3d34-4d9c-ad37-3369e6077cd0","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/2aa4bf78-546d-4846-89d3-20642e8e49b2 {"id":"2aa4bf78-546d-4846-89d3-20642e8e49b2","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f901f2a 2aa4bf78-546d-4846-89d3-20642e8e49b2 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f901f53 0a732fa0-3d34-4d9c-ad37-3369e6077cd0 /tidb/cdc/default/default/upstream/7374684810041326586 {"id":7374684810041326586,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0a732fa0-3d34-4d9c-ad37-3369e6077cd0 {"id":"0a732fa0-3d34-4d9c-ad37-3369e6077cd0","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/2aa4bf78-546d-4846-89d3-20642e8e49b2 {"id":"2aa4bf78-546d-4846-89d3-20642e8e49b2","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f901f2a 2aa4bf78-546d-4846-89d3-20642e8e49b2 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f901f53 0a732fa0-3d34-4d9c-ad37-3369e6077cd0 /tidb/cdc/default/default/upstream/7374684810041326586 {"id":7374684810041326586,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0a732fa0-3d34-4d9c-ad37-3369e6077cd0 {"id":"0a732fa0-3d34-4d9c-ad37-3369e6077cd0","address":"127.0.0.1:8302","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/capture/2aa4bf78-546d-4846-89d3-20642e8e49b2 {"id":"2aa4bf78-546d-4846-89d3-20642e8e49b2","address":"127.0.0.1:8301","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f901f2a 2aa4bf78-546d-4846-89d3-20642e8e49b2 /tidb/cdc/default/__cdc_meta__/owner/22318fc84f901f53 0a732fa0-3d34-4d9c-ad37-3369e6077cd0 /tidb/cdc/default/default/upstream/7374684810041326586 {"id":7374684810041326586,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Thu May 30 15:03:53 CST 2024] <<<<<< START Pulsar consumer in changefeed_auto_stop case >>>>>> check_changefeed_state http://127.0.0.1:2379 1097f59d-3d03-491e-af40-6871b0d4cee6 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=1097f59d-3d03-491e-af40-6871b0d4cee6 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 1097f59d-3d03-491e-af40-6871b0d4cee6 -s + info='{ "upstream_id": 7374684810041326586, "namespace": "default", "id": "1097f59d-3d03-491e-af40-6871b0d4cee6", "state": "normal", "checkpoint_tso": 450115043353690113, "checkpoint_time": "2024-05-30 15:03:45.098", "error": null }' + echo '{ "upstream_id": 7374684810041326586, "namespace": "default", "id": "1097f59d-3d03-491e-af40-6871b0d4cee6", "state": "normal", "checkpoint_tso": 450115043353690113, "checkpoint_time": "2024-05-30 15:03:45.098", "error": null }' { "upstream_id": 7374684810041326586, "namespace": "default", "id": "1097f59d-3d03-491e-af40-6871b0d4cee6", "state": "normal", "checkpoint_tso": 450115043353690113, "checkpoint_time": "2024-05-30 15:03:45.098", "error": null } ++ echo '{' '"upstream_id":' 7374684810041326586, '"namespace":' '"default",' '"id":' '"1097f59d-3d03-491e-af40-6871b0d4cee6",' '"state":' '"normal",' '"checkpoint_tso":' 450115043353690113, '"checkpoint_time":' '"2024-05-30' '15:03:45.098",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7374684810041326586, '"namespace":' '"default",' '"id":' '"1097f59d-3d03-491e-af40-6871b0d4cee6",' '"state":' '"normal",' '"checkpoint_tso":' 450115043353690113, '"checkpoint_time":' '"2024-05-30' '15:03:45.098",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully table changefeed_auto_stop_1.usertable not exists for 1-th check, retry later check diff failed 3-th time, retry later table changefeed_auto_stop_1.usertable exists table changefeed_auto_stop_2.usertable not exists for 1-th check, retry later check diff successfully table changefeed_auto_stop_2.usertable exists table changefeed_auto_stop_3.usertable not exists for 1-th check, retry later [2024/05/30 15:03:57.830 +08:00] [INFO] [main.go:61] ["table mover started"] [2024/05/30 15:03:57.832 +08:00] [INFO] [main.go:166] ["new cluster initialized"] [2024/05/30 15:03:57.833 +08:00] [DEBUG] [main.go:192] ["retrieved owner ID"] [ownerID=6f77f2cc-40d7-4c08-95fc-e9e78b98b116] [2024/05/30 15:03:57.833 +08:00] [DEBUG] [main.go:199] ["retrieved owner addr"] [ownerAddr=127.0.0.1:8300] [2024/05/30 15:03:57.833 +08:00] [DEBUG] [main.go:210] ["retrieved changefeeds"] [changefeedsError="json: unsupported type: map[model.ChangeFeedID]*mvccpb.KeyValue"] [2024/05/30 15:03:58.027 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=e32af784-e449-4e4a-aa5d-f9dbc2f3a33f] [captureID=1bce962d-74d3-40b8-9faa-f543f8937971] [processorDetail="{\"table_ids\":[]}"] [2024/05/30 15:03:58.227 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=e32af784-e449-4e4a-aa5d-f9dbc2f3a33f] [captureID=6f77f2cc-40d7-4c08-95fc-e9e78b98b116] [processorDetail="{\"table_ids\":[106,108]}"] [2024/05/30 15:03:58.427 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=e32af784-e449-4e4a-aa5d-f9dbc2f3a33f] [captureID=cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8] [processorDetail="{\"table_ids\":[106]}"] [2024/05/30 15:03:58.427 +08:00] [INFO] [main.go:75] ["task status"] [status="{\"1bce962d-74d3-40b8-9faa-f543f8937971\":[],\"6f77f2cc-40d7-4c08-95fc-e9e78b98b116\":[{\"ID\":106,\"Changefeed\":\"e32af784-e449-4e4a-aa5d-f9dbc2f3a33f\"},{\"ID\":108,\"Changefeed\":\"e32af784-e449-4e4a-aa5d-f9dbc2f3a33f\"}],\"cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8\":[{\"ID\":106,\"Changefeed\":\"e32af784-e449-4e4a-aa5d-f9dbc2f3a33f\"}]}"] [2024/05/30 15:03:58.428 +08:00] [DEBUG] [main.go:288] ["preparing HTTP API call to owner"] [formStr="cf-id=e32af784-e449-4e4a-aa5d-f9dbc2f3a33f&target-cp-id=cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8&table-id=106"] [2024/05/30 15:03:58.477 +08:00] [INFO] [main.go:180] ["moved table successful"] [tableID=106] [2024/05/30 15:03:58.477 +08:00] [DEBUG] [main.go:288] ["preparing HTTP API call to owner"] [formStr="cf-id=e32af784-e449-4e4a-aa5d-f9dbc2f3a33f&target-cp-id=cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8&table-id=108"] [2024/05/30 15:03:58.527 +08:00] [INFO] [main.go:180] ["moved table successful"] [tableID=108] [2024/05/30 15:03:58.527 +08:00] [INFO] [main.go:114] ["all tables are moved"] [sourceCapture=6f77f2cc-40d7-4c08-95fc-e9e78b98b116] [targetCapture=cf1b30b1-22c9-4fdd-a0c8-27fc8699b4e8] check diff successfully table move_table.check2 not exists for 1-th check, retry later table changefeed_auto_stop_3.usertable exists table changefeed_auto_stop_4.usertable not exists for 1-th check, retry later table move_table.check2 exists check diff successfully wait process cdc.test exit for 1-th time... table changefeed_auto_stop_4.usertable not exists for 2-th check, retry later wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:04:02 CST 2024] <<<<<< run test case move_table success! >>>>>> table changefeed_auto_stop_4.usertable exists check diff failed 1-th time, retry later check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 15:04:07 CST 2024] <<<<<< run test case changefeed_auto_stop success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/autorandom/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/autorandom Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Verifying downstream PD is started... \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2141dc2c0009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-179cx, pid:27719, start at 2024-05-30 15:04:27.668366496 +0800 CST m=+5.412051553 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:06:27.677 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:04:27.659 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:54:27.659 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2141dc2c0009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-179cx, pid:27719, start at 2024-05-30 15:04:27.668366496 +0800 CST m=+5.412051553 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:06:27.677 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:04:27.659 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:54:27.659 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2141dddc000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-179cx, pid:27798, start at 2024-05-30 15:04:27.778964615 +0800 CST m=+5.456804028 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:06:27.789 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:04:27.767 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:54:27.767 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/autorandom/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/autorandom/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Thu May 30 15:04:32 CST 2024] <<<<<< START cdc server in autorandom case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.autorandom.2909129093.out server --log-file /tmp/tidb_cdc_test/autorandom/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/autorandom/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 *************************** 1. row *************************** count(distinct region_id): 1 table region_merge.t1 exists check diff failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:04:35 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a0009328-ba56-46b7-a043-c650252dc294 {"id":"a0009328-ba56-46b7-a043-c650252dc294","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc85049e6c1 a0009328-ba56-46b7-a043-c650252dc294 /tidb/cdc/default/default/upstream/7374685004664806793 {"id":7374685004664806793,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a0009328-ba56-46b7-a043-c650252dc294 {"id":"a0009328-ba56-46b7-a043-c650252dc294","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc85049e6c1 a0009328-ba56-46b7-a043-c650252dc294 /tidb/cdc/default/default/upstream/7374685004664806793 {"id":7374685004664806793,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a0009328-ba56-46b7-a043-c650252dc294 {"id":"a0009328-ba56-46b7-a043-c650252dc294","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc85049e6c1 a0009328-ba56-46b7-a043-c650252dc294 /tidb/cdc/default/default/upstream/7374685004664806793 {"id":7374685004664806793,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: 90c734df-96f5-419a-aaaa-112c11f1c926 Info: {"upstream_id":7374685004664806793,"namespace":"default","id":"90c734df-96f5-419a-aaaa-112c11f1c926","sink_uri":"pulsar://127.0.0.1:6650/ticdc-autorandom-test-26743?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T15:04:36.002442314+08:00","start_ts":450115056647274501,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115056647274501,"checkpoint_ts":450115056647274501,"checkpoint_time":"2024-05-30 15:04:35.809"} [Thu May 30 15:04:36 CST 2024] <<<<<< START Pulsar consumer in autorandom case >>>>>> check diff successfully table autorandom_test.table_a not exists for 1-th check, retry later wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:04:39 CST 2024] <<<<<< run test case region_merge success! >>>>>> table autorandom_test.table_a exists check diff failed 1-th time, retry later check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Thu May 30 15:04:43 CST 2024] <<<<<< run test case autorandom success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/common_1/run.sh using Sink-Type: pulsar... <<================= \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/generate_column/run.sh using Sink-Type: pulsar... <<================= ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 1662 0 --:--:-- --:--:-- --:--:-- 1661 100 221 100 221 0 0 1660 0 --:--:-- --:--:-- --:--:-- 1661 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-30 15:04:41.754","puller_resolved_ts":"2024-05-30 15:04:33.754","last_synced_ts":"2024-05-30 15:02:24.904","now_ts":"2024-05-30 15:04:42.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:04:41.754","puller_resolved_ts":"2024-05-30' '15:04:33.754","last_synced_ts":"2024-05-30' '15:02:24.904","now_ts":"2024-05-30' '15:04:42.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' + kill_pd ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 14624 8.4 0.0 13509836 138796 ? Sl 15:02 0:13 pd-server --advertise-client-urls http://127.0.0.1:2379 --client-urls http://0.0.0.0:2379 --advertise-peer-urls http://127.0.0.1:2380 --peer-urls http://0.0.0.0:2380 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/pd1.log --data-dir /tmp/tidb_cdc_test/synced_status/pd1 --name=pd1 --initial-cluster=pd1=http://127.0.0.1:2380 jenkins 14680 5.6 0.0 13288576 136168 ? Sl 15:02 0:08 pd-server --advertise-client-urls http://127.0.0.1:2479 --client-urls http://0.0.0.0:2479 --advertise-peer-urls http://127.0.0.1:2480 --peer-urls http://0.0.0.0:2480 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/down_pd.log --data-dir /tmp/tidb_cdc_test/synced_status/down_pd' ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 + sleep 20 {"level":"warn","ts":1717052687.8871777,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002f89a40/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1717052687.8872757,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1717052687.8971255,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0036e4fc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1717052687.8972087,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1717052688.781384,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002508a80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1717052688.7814472,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-05-30T15:04:52.617603+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0008f1340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:04:52.634909+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004f9180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:04:52.70549+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f42e00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } {"level":"warn","ts":"2024-05-30T15:04:58.619187+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0008f1340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:04:58.636252+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004f9180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:04:58.706562+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f42e00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:05:04.620389+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0008f1340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:05:04.637547+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004f9180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:05:04.708298+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f42e00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:05:10.621596+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0008f1340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:05:10.638726+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004f9180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:05:10.709358+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f42e00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:05:12.609677+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0008f1340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2024-05-30T15:05:12.609759+0800","logger":"etcd-client","caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-05-30T15:05:12.625819+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004f9180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2024-05-30T15:05:12.62588+0800","logger":"etcd-client","caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-05-30T15:05:12.691035+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f42e00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":"2024-05-30T15:05:12.691098+0800","logger":"etcd-client","caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:14 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:05:16.622516+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0008f1340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:05:16.639869+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004f9180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:05:16.710525+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f42e00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:15 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:17 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:18 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:05:22.623256+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0008f1340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:05:22.641303+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004f9180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:05:22.711905+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f42e00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"warn","ts":1717052722.8891804,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002f89a40/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1717052722.8892632,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1717052722.8986444,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0036e4fc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1717052722.8987267,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:21 --:--:-- 0{"level":"warn","ts":1717052723.7821066,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002508a80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1717052723.7821717,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:22 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:23 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:24 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:25 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:26 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:05:28.623959+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0008f1340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:05:28.64273+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004f9180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:05:28.712276+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f42e00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} check_changefeed_state http://127.0.0.1:2379 e7f464f1-4b3a-484c-a019-356157a798f7 finished null + endpoints=http://127.0.0.1:2379 + changefeed_id=e7f464f1-4b3a-484c-a019-356157a798f7 + expected_state=finished + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c e7f464f1-4b3a-484c-a019-356157a798f7 -s + info='{ "upstream_id": 7374684812134114084, "namespace": "default", "id": "e7f464f1-4b3a-484c-a019-356157a798f7", "state": "finished", "checkpoint_tso": 450115067831386118, "checkpoint_time": "2024-05-30 15:05:18.473", "error": null }' + echo '{ "upstream_id": 7374684812134114084, "namespace": "default", "id": "e7f464f1-4b3a-484c-a019-356157a798f7", "state": "finished", "checkpoint_tso": 450115067831386118, "checkpoint_time": "2024-05-30 15:05:18.473", "error": null }' { "upstream_id": 7374684812134114084, "namespace": "default", "id": "e7f464f1-4b3a-484c-a019-356157a798f7", "state": "finished", "checkpoint_tso": 450115067831386118, "checkpoint_time": "2024-05-30 15:05:18.473", "error": null } ++ echo '{' '"upstream_id":' 7374684812134114084, '"namespace":' '"default",' '"id":' '"e7f464f1-4b3a-484c-a019-356157a798f7",' '"state":' '"finished",' '"checkpoint_tso":' 450115067831386118, '"checkpoint_time":' '"2024-05-30' '15:05:18.473",' '"error":' null '}' ++ jq -r .state + state=finished + [[ ! finished == \f\i\n\i\s\h\e\d ]] ++ echo '{' '"upstream_id":' 7374684812134114084, '"namespace":' '"default",' '"id":' '"e7f464f1-4b3a-484c-a019-356157a798f7",' '"state":' '"finished",' '"checkpoint_tso":' 450115067831386118, '"checkpoint_time":' '"2024-05-30' '15:05:18.473",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... wait process cdc.test exit for 4-th time... wait process cdc.test exit for 5-th time... wait process cdc.test exit for 6-th time... wait process cdc.test exit for 7-th time... wait process cdc.test exit for 8-th time... cdc.test: no process found wait process cdc.test exit for 9-th time... process cdc.test already exit [Thu May 30 15:05:30 CST 2024] <<<<<< run test case changefeed_finish success! >>>>>> 0 0 0 0 0 0 0 0 --:--:-- 0:00:27 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:28 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 27 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 33 + synced_status='{ "error_msg": "[CDC:ErrPDEtcdAPIError]etcd api call error: context deadline exceeded", "error_code": "CDC:ErrPDEtcdAPIError" }' ++ jq -r .error_code ++ echo '{' '"error_msg":' '"[CDC:ErrPDEtcdAPIError]etcd' api call error: context deadline 'exceeded",' '"error_code":' '"CDC:ErrPDEtcdAPIError"' '}' + error_code=CDC:ErrPDEtcdAPIError + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/force_replicate_table/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... + run_case_with_unavailable_tikv conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/force_replicate_table Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2147258c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-0gwt8, pid:6890, start at 2024-05-30 15:05:54.287072205 +0800 CST m=+5.411495338 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:07:54.294 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:05:54.275 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:55:54.275 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2147258c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-0gwt8, pid:6890, start at 2024-05-30 15:05:54.287072205 +0800 CST m=+5.411495338 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:07:54.294 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:05:54.275 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:55:54.275 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214727080009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-0gwt8, pid:6955, start at 2024-05-30 15:05:54.379514801 +0800 CST m=+5.436605443 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:07:54.389 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:05:54.370 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:55:54.370 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/force_replicate_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/force_replicate_table/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2147419c0004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:17858, start at 2024-05-30 15:05:56.074317298 +0800 CST m=+5.205587812 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:07:56.083 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:05:56.071 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:55:56.071 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2147419c0004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:17858, start at 2024-05-30 15:05:56.074317298 +0800 CST m=+5.205587812 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:07:56.083 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:05:56.071 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:55:56.071 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f214743240014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:17927, start at 2024-05-30 15:05:56.198843689 +0800 CST m=+5.263753867 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:07:56.206 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:05:56.169 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:55:56.169 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.19222.out cli tso query --pd=http://127.0.0.1:2379 [Thu May 30 15:05:59 CST 2024] <<<<<< START cdc server in force_replicate_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.force_replicate_table.82168218.out server --log-file /tmp/tidb_cdc_test/force_replicate_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/force_replicate_table/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x + tso='450115078591610881 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115078591610881 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=450115078591610881 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Thu May 30 15:06:00 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.1926119263.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:06:02 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1394ab77-7cbd-4ef5-95c2-429b9c208dd3 {"id":"1394ab77-7cbd-4ef5-95c2-429b9c208dd3","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8519e66be 1394ab77-7cbd-4ef5-95c2-429b9c208dd3 /tidb/cdc/default/default/upstream/7374685388930851044 {"id":7374685388930851044,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1394ab77-7cbd-4ef5-95c2-429b9c208dd3 {"id":"1394ab77-7cbd-4ef5-95c2-429b9c208dd3","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8519e66be 1394ab77-7cbd-4ef5-95c2-429b9c208dd3 /tidb/cdc/default/default/upstream/7374685388930851044 {"id":7374685388930851044,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1394ab77-7cbd-4ef5-95c2-429b9c208dd3 {"id":"1394ab77-7cbd-4ef5-95c2-429b9c208dd3","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc8519e66be 1394ab77-7cbd-4ef5-95c2-429b9c208dd3 /tidb/cdc/default/default/upstream/7374685388930851044 {"id":7374685388930851044,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: 75aa6523-e4d0-4661-8a53-38fe2cacfb89 Info: {"upstream_id":7374685388930851044,"namespace":"default","id":"75aa6523-e4d0-4661-8a53-38fe2cacfb89","sink_uri":"pulsar://127.0.0.1:6650/ticdc-force_replicate_table-test-23248?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-30T15:06:02.693559724+08:00","start_ts":450115078540492801,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":true,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115078540492801,"checkpoint_ts":450115078540492801,"checkpoint_time":"2024-05-30 15:05:59.325"} [Thu May 30 15:06:02 CST 2024] <<<<<< START Pulsar consumer in force_replicate_table case >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:06:04 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/59bc7418-52f5-403e-a035-813e94b05b8d {"id":"59bc7418-52f5-403e-a035-813e94b05b8d","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc851a43dbf 59bc7418-52f5-403e-a035-813e94b05b8d /tidb/cdc/default/default/upstream/7374685395827110814 {"id":7374685395827110814,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/59bc7418-52f5-403e-a035-813e94b05b8d {"id":"59bc7418-52f5-403e-a035-813e94b05b8d","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc851a43dbf 59bc7418-52f5-403e-a035-813e94b05b8d /tidb/cdc/default/default/upstream/7374685395827110814 {"id":7374685395827110814,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/59bc7418-52f5-403e-a035-813e94b05b8d {"id":"59bc7418-52f5-403e-a035-813e94b05b8d","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc851a43dbf 59bc7418-52f5-403e-a035-813e94b05b8d /tidb/cdc/default/default/upstream/7374685395827110814 {"id":7374685395827110814,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=450115078591610881 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.19305.out cli changefeed create --start-ts=450115078591610881 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7374685395827110814,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T15:06:04.470307826+08:00","start_ts":450115078591610881,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115078591610881,"checkpoint_ts":450115078591610881,"checkpoint_time":"2024-05-30 15:05:59.520"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 table force_replicate_table.t0 exists table force_replicate_table.t1 exists table force_replicate_table.t2 exists table force_replicate_table.t3 not exists for 1-th check, retry later table force_replicate_table.t3 exists table force_replicate_table.t4 not exists for 1-th check, retry later + kill_tikv ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 17292 26.6 0.4 3812992 1731404 ? Sl 15:05 0:06 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20160 --status-addr 127.0.0.1:20181 --log-file /tmp/tidb_cdc_test/synced_status/tikv1.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv1 jenkins 17293 18.1 0.4 3785856 1680680 ? Sl 15:05 0:04 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20161 --status-addr 127.0.0.1:20182 --log-file /tmp/tidb_cdc_test/synced_status/tikv2.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv2 jenkins 17294 18.3 0.4 3779716 1661068 ? Sl 15:05 0:04 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20162 --status-addr 127.0.0.1:20183 --log-file /tmp/tidb_cdc_test/synced_status/tikv3.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv3 jenkins 17296 25.2 0.4 3804800 1721116 ? Sl 15:05 0:06 tikv-server --pd 127.0.0.1:2479 -A 127.0.0.1:21160 --status-addr 127.0.0.1:21180 --log-file /tmp/tidb_cdc_test/synced_status/tikv_down.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv_down' ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 2736 0 --:--:-- --:--:-- --:--:-- 2761 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-30 15:06:11.920","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"2024-05-30 15:06:05.971","now_ts":"2024-05-30 15:06:13.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:06:11.920","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-05-30' '15:06:05.971","now_ts":"2024-05-30' '15:06:13.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:06:11.920","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-05-30' '15:06:05.971","now_ts":"2024-05-30' '15:06:13.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 table force_replicate_table.t4 exists table force_replicate_table.t5 not exists for 1-th check, retry later table force_replicate_table.t5 exists table force_replicate_table.t6 not exists for 1-th check, retry later table force_replicate_table.t6 not exists for 2-th check, retry later table force_replicate_table.t6 exists check_data_subset force_replicate_table.t0 127.0.0.1 4000 127.0.0.1 3306 run task successfully check_data_subset force_replicate_table.t1 127.0.0.1 4000 127.0.0.1 3306 run task successfully check_data_subset force_replicate_table.t2 127.0.0.1 4000 127.0.0.1 3306 run task successfully check_data_subset force_replicate_table.t3 127.0.0.1 4000 127.0.0.1 3306 run task successfully check_data_subset force_replicate_table.t4 127.0.0.1 4000 127.0.0.1 3306 run task successfully check_data_subset force_replicate_table.t5 127.0.0.1 4000 127.0.0.1 3306 run task successfully check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 id=19,a=NULL doesn't exist in downstream table force_replicate_table.t6 run task failed 1-th time, retry later check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Thu May 30 15:06:28 CST 2024] <<<<<< run test case force_replicate_table success! >>>>>> \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 8409 0 --:--:-- --:--:-- --:--:-- 8505 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-30 15:06:12.920","puller_resolved_ts":"2024-05-30 15:06:12.920","last_synced_ts":"2024-05-30 15:06:05.971","now_ts":"2024-05-30 15:08:23.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:06:12.920","puller_resolved_ts":"2024-05-30' '15:06:12.920","last_synced_ts":"2024-05-30' '15:06:05.971","now_ts":"2024-05-30' '15:08:23.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ jq -r .info ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:06:12.920","puller_resolved_ts":"2024-05-30' '15:06:12.920","last_synced_ts":"2024-05-30' '15:06:05.971","now_ts":"2024-05-30' '15:08:23.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tidb conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2151b17c0009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:20449, start at 2024-05-30 15:08:47.080591322 +0800 CST m=+5.301764011 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:10:47.088 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:08:47.071 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:58:47.071 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2151b17c0009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:20449, start at 2024-05-30 15:08:47.080591322 +0800 CST m=+5.301764011 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:10:47.088 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:08:47.071 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:58:47.071 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2151b1a00013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:20511, start at 2024-05-30 15:08:47.114484264 +0800 CST m=+5.271715101 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:10:47.124 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:08:47.129 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-14:58:47.129 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.21738.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='450115123916570625 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115123916570625 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=450115123916570625 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Thu May 30 15:08:53 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2176521767.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:08:56 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cb11c90a-69b0-43d3-a46e-fa89c07c4a43 {"id":"cb11c90a-69b0-43d3-a46e-fa89c07c4a43","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc85441c7c3 cb11c90a-69b0-43d3-a46e-fa89c07c4a43 /tidb/cdc/default/default/upstream/7374686130898016630 {"id":7374686130898016630,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cb11c90a-69b0-43d3-a46e-fa89c07c4a43 {"id":"cb11c90a-69b0-43d3-a46e-fa89c07c4a43","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc85441c7c3 cb11c90a-69b0-43d3-a46e-fa89c07c4a43 /tidb/cdc/default/default/upstream/7374686130898016630 {"id":7374686130898016630,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cb11c90a-69b0-43d3-a46e-fa89c07c4a43 {"id":"cb11c90a-69b0-43d3-a46e-fa89c07c4a43","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc85441c7c3 cb11c90a-69b0-43d3-a46e-fa89c07c4a43 /tidb/cdc/default/default/upstream/7374686130898016630 {"id":7374686130898016630,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=450115123916570625 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.21820.out cli changefeed create --start-ts=450115123916570625 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7374686130898016630,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T15:08:57.346299344+08:00","start_ts":450115123916570625,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115123916570625,"checkpoint_ts":450115123916570625,"checkpoint_time":"2024-05-30 15:08:52.421"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 + kill_tidb ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 20449 12.4 0.0 2426964 190444 ? Sl 15:08 0:02 tidb-server -P 4000 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1717052921772249044.toml --store tikv --path 127.0.0.1:2379 --status=10080 --log-file /tmp/tidb_cdc_test/synced_status/tidb.log jenkins 20453 3.8 0.0 2048220 164076 ? Sl 15:08 0:00 tidb-server -P 4001 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1717052921775339377.toml --store tikv --path 127.0.0.1:2379 --status=10081 --log-file /tmp/tidb_cdc_test/synced_status/tidb_other.log jenkins 20511 12.5 0.0 2501272 216460 ? Sl 15:08 0:03 tidb-server -P 3306 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1717052921836168480.toml --store tikv --path 127.0.0.1:2479 --status=20080 --log-file /tmp/tidb_cdc_test/synced_status/tidb_down.log' ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 2289 0 --:--:-- --:--:-- --:--:-- 2292 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-30 15:09:04.821","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"2024-05-30 15:08:58.872","now_ts":"2024-05-30 15:09:06.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:09:04.821","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-05-30' '15:08:58.872","now_ts":"2024-05-30' '15:09:06.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:09:04.821","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-05-30' '15:08:58.872","now_ts":"2024-05-30' '15:09:06.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 3263 0 --:--:-- --:--:-- --:--:-- 3250 100 221 100 221 0 0 3258 0 --:--:-- --:--:-- --:--:-- 3250 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-30 15:11:14.971","puller_resolved_ts":"2024-05-30 15:11:07.971","last_synced_ts":"2024-05-30 15:08:58.872","now_ts":"2024-05-30 15:11:16.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:11:14.971","puller_resolved_ts":"2024-05-30' '15:11:07.971","last_synced_ts":"2024-05-30' '15:08:58.872","now_ts":"2024-05-30' '15:11:16.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:11:14.971","puller_resolved_ts":"2024-05-30' '15:11:07.971","last_synced_ts":"2024-05-30' '15:08:58.872","now_ts":"2024-05-30' '15:11:16.000","info":"Data' syncing is 'finished"}' ++ jq -r .info + info='Data syncing is finished' + target_message='Data syncing is finished' + '[' 'Data syncing is finished' '!=' 'Data syncing is finished' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_failpoint conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f215c3afc0019 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:22886, start at 2024-05-30 15:11:39.741766519 +0800 CST m=+5.240922927 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:13:39.752 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:11:39.711 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:01:39.711 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f215c3afc0019 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:22886, start at 2024-05-30 15:11:39.741766519 +0800 CST m=+5.240922927 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:13:39.752 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:11:39.711 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:01:39.711 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f215c3c740014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:22971, start at 2024-05-30 15:11:39.832764436 +0800 CST m=+5.245306619 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:13:39.840 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:11:39.805 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:01:39.805 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status + export 'GO_FAILPOINTS=github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.24264.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='450115168675037185 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115168675037185 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=450115168675037185 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Thu May 30 15:11:44 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2429624298.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:11:47 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/88713910-eeee-4933-8522-43f48946d973 {"id":"88713910-eeee-4933-8522-43f48946d973","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc856e488be 88713910-eeee-4933-8522-43f48946d973 /tidb/cdc/default/default/upstream/7374686874708727137 {"id":7374686874708727137,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/88713910-eeee-4933-8522-43f48946d973 {"id":"88713910-eeee-4933-8522-43f48946d973","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc856e488be 88713910-eeee-4933-8522-43f48946d973 /tidb/cdc/default/default/upstream/7374686874708727137 {"id":7374686874708727137,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/88713910-eeee-4933-8522-43f48946d973 {"id":"88713910-eeee-4933-8522-43f48946d973","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc856e488be 88713910-eeee-4933-8522-43f48946d973 /tidb/cdc/default/default/upstream/7374686874708727137 {"id":7374686874708727137,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=450115168675037185 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.24345.out cli changefeed create --start-ts=450115168675037185 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7374686874708727137,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T15:11:48.064620794+08:00","start_ts":450115168675037185,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115168675037185,"checkpoint_ts":450115168675037185,"checkpoint_time":"2024-05-30 15:11:43.161"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x + sleep 20 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 8261 0 --:--:-- --:--:-- --:--:-- 8310 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-30 15:11:43.161","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-05-30 15:12:09.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ jq .synced ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:11:43.161","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-30' '15:12:09.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:11:43.161","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-30' '15:12:09.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + export GO_FAILPOINTS= + GO_FAILPOINTS= + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_normal_case_and_unavailable_pd conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f215f98900014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:25343, start at 2024-05-30 15:12:34.895498092 +0800 CST m=+5.140884349 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:14:34.901 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:12:34.903 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:02:34.903 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f215f98900014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:25343, start at 2024-05-30 15:12:34.895498092 +0800 CST m=+5.140884349 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:14:34.901 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:12:34.903 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:02:34.903 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f215f9a0c0015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:25423, start at 2024-05-30 15:12:34.990049794 +0800 CST m=+5.184734875 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:14:34.998 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:12:34.997 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:02:34.997 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.26647.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='450115183156396033 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115183156396033 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=450115183156396033 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Thu May 30 15:12:39 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2667426676.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:12:42 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b2bb789b-dbc8-4486-98ee-9d482d48508a {"id":"b2bb789b-dbc8-4486-98ee-9d482d48508a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc857ba60c0 b2bb789b-dbc8-4486-98ee-9d482d48508a /tidb/cdc/default/default/upstream/7374687109960385485 {"id":7374687109960385485,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b2bb789b-dbc8-4486-98ee-9d482d48508a {"id":"b2bb789b-dbc8-4486-98ee-9d482d48508a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc857ba60c0 b2bb789b-dbc8-4486-98ee-9d482d48508a /tidb/cdc/default/default/upstream/7374687109960385485 {"id":7374687109960385485,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b2bb789b-dbc8-4486-98ee-9d482d48508a {"id":"b2bb789b-dbc8-4486-98ee-9d482d48508a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc857ba60c0 b2bb789b-dbc8-4486-98ee-9d482d48508a /tidb/cdc/default/default/upstream/7374687109960385485 {"id":7374687109960385485,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=450115183156396033 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.26727.out cli changefeed create --start-ts=450115183156396033 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7374687109960385485,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T15:12:43.346722252+08:00","start_ts":450115183156396033,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115183156396033,"checkpoint_ts":450115183156396033,"checkpoint_time":"2024-05-30 15:12:38.403"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2049 0 --:--:-- --:--:-- --:--:-- 2065 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-30 15:12:38.403","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-05-30 15:12:44.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:12:38.403","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-30' '15:12:44.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:12:38.403","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-30' '15:12:44.000","info":"Data' syncing is 'finished"}' ++ jq -r .sink_checkpoint_ts + sink_checkpoint_ts='2024-05-30 15:12:38.403' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:12:38.403","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-30' '15:12:44.000","info":"Data' syncing is 'finished"}' ++ jq -r .puller_resolved_ts + puller_resolved_ts='1970-01-01 08:00:00.000' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:12:38.403","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-30' '15:12:44.000","info":"Data' syncing is 'finished"}' ++ jq -r .last_synced_ts + last_synced_ts='1970-01-01 08:00:00.000' + '[' true '!=' true ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' ++ date '+%Y-%m-%d %H:%M:%S' + current='2024-05-30 15:12:44' + echo 'sink_checkpoint_ts is 2024-05-30' 15:12:38.403 sink_checkpoint_ts is 2024-05-30 15:12:38.403 ++ date -d '2024-05-30 15:12:38.403' +%s + checkpoint_timestamp=1717053158 ++ date -d '2024-05-30 15:12:44' +%s + current_timestamp=1717053164 + '[' 6 -gt 300 ']' + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 2340 0 --:--:-- --:--:-- --:--:-- 2359 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-30 15:12:50.803","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"2024-05-30 15:12:45.503","now_ts":"2024-05-30 15:12:52.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:12:50.803","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-05-30' '15:12:45.503","now_ts":"2024-05-30' '15:12:52.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:12:50.803","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-05-30' '15:12:45.503","now_ts":"2024-05-30' '15:12:52.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2639 0 --:--:-- --:--:-- --:--:-- 2662 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-30 15:15:00.953","puller_resolved_ts":"2024-05-30 15:14:54.953","last_synced_ts":"2024-05-30 15:12:45.503","now_ts":"2024-05-30 15:15:02.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:15:00.953","puller_resolved_ts":"2024-05-30' '15:14:54.953","last_synced_ts":"2024-05-30' '15:12:45.503","now_ts":"2024-05-30' '15:15:02.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' + kill_pd ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 24612 8.1 0.0 13690116 139852 ? Sl 15:12 0:13 pd-server --advertise-client-urls http://127.0.0.1:2379 --client-urls http://0.0.0.0:2379 --advertise-peer-urls http://127.0.0.1:2380 --peer-urls http://0.0.0.0:2380 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/pd1.log --data-dir /tmp/tidb_cdc_test/synced_status/pd1 --name=pd1 --initial-cluster=pd1=http://127.0.0.1:2380 jenkins 24669 5.5 0.0 13501320 134796 ? Sl 15:12 0:08 pd-server --advertise-client-urls http://127.0.0.1:2479 --client-urls http://0.0.0.0:2479 --advertise-peer-urls http://127.0.0.1:2480 --peer-urls http://0.0.0.0:2480 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/down_pd.log --data-dir /tmp/tidb_cdc_test/synced_status/down_pd' ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 + sleep 20 {"level":"warn","ts":1717053309.9247372,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc003a77dc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1717053309.9248288,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1717053309.9968238,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0022c6000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1717053309.9969025,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1717053310.025369,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc003a6e380/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1717053310.0254514,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-05-30T15:15:14.816034+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f64700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:15:14.817042+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000eaefc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:15:14.902122+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000d19500/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:15:20.81688+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f64700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:15:20.818134+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000eaefc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:15:20.903161+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000d19500/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:15:26.817518+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f64700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:15:26.81927+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000eaefc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:15:26.905957+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000d19500/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:15:32.818664+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f64700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:15:32.820268+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000eaefc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:15:32.907579+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000d19500/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:15:34.805657+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f64700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2024-05-30T15:15:34.805714+0800","logger":"etcd-client","caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-05-30T15:15:34.806705+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000eaefc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2024-05-30T15:15:34.806758+0800","logger":"etcd-client","caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:15:34.892005+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000d19500/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":"2024-05-30T15:15:34.892074+0800","logger":"etcd-client","caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:14 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:15 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:15:38.819642+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f64700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:15:38.821353+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000eaefc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:15:38.908616+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000d19500/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:17 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:18 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:21 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:15:44.82065+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f64700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:15:44.823186+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000eaefc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:22 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:15:44.909286+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000d19500/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"warn","ts":1717053344.9262486,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc003a77dc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1717053344.9263022,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1717053344.9976008,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0022c6000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1717053344.9976697,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1717053345.0266452,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc003a6e380/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1717053345.0267496,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:23 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:24 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:25 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:26 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:27 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:15:50.821194+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f64700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-30T15:15:50.82397+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000eaefc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:28 --:--:-- 0{"level":"warn","ts":"2024-05-30T15:15:50.910301+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000d19500/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 27 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 33 + synced_status='{ "error_msg": "[CDC:ErrPDEtcdAPIError]etcd api call error: context deadline exceeded", "error_code": "CDC:ErrPDEtcdAPIError" }' ++ jq -r .error_code ++ echo '{' '"error_msg":' '"[CDC:ErrPDEtcdAPIError]etcd' api call error: context deadline 'exceeded",' '"error_code":' '"CDC:ErrPDEtcdAPIError"' '}' + error_code=CDC:ErrPDEtcdAPIError + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tikv conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f216cfc000004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:27904, start at 2024-05-30 15:16:14.211236865 +0800 CST m=+5.135224459 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:18:14.217 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:16:14.208 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:06:14.208 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f216cfc000004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:27904, start at 2024-05-30 15:16:14.211236865 +0800 CST m=+5.135224459 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:18:14.217 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:16:14.208 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:06:14.208 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f216cfd740017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:27978, start at 2024-05-30 15:16:14.347835705 +0800 CST m=+5.219246925 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:18:14.355 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:16:14.351 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:06:14.351 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.29220.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='450115240645885953 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115240645885953 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=450115240645885953 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Thu May 30 15:16:19 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2924729249.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:16:22 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/34d3f414-bb8e-4ab2-b1d3-5e3962407d20 {"id":"34d3f414-bb8e-4ab2-b1d3-5e3962407d20","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc85b18e3c2 34d3f414-bb8e-4ab2-b1d3-5e3962407d20 /tidb/cdc/default/default/upstream/7374688047970823113 {"id":7374688047970823113,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/34d3f414-bb8e-4ab2-b1d3-5e3962407d20 {"id":"34d3f414-bb8e-4ab2-b1d3-5e3962407d20","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc85b18e3c2 34d3f414-bb8e-4ab2-b1d3-5e3962407d20 /tidb/cdc/default/default/upstream/7374688047970823113 {"id":7374688047970823113,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/34d3f414-bb8e-4ab2-b1d3-5e3962407d20 {"id":"34d3f414-bb8e-4ab2-b1d3-5e3962407d20","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc85b18e3c2 34d3f414-bb8e-4ab2-b1d3-5e3962407d20 /tidb/cdc/default/default/upstream/7374688047970823113 {"id":7374688047970823113,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=450115240645885953 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.29296.out cli changefeed create --start-ts=450115240645885953 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7374688047970823113,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T15:16:22.721342791+08:00","start_ts":450115240645885953,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115240645885953,"checkpoint_ts":450115240645885953,"checkpoint_time":"2024-05-30 15:16:17.708"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 + kill_tikv ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 27296 17.8 0.4 3786372 1689104 ? Sl 15:16 0:04 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20160 --status-addr 127.0.0.1:20181 --log-file /tmp/tidb_cdc_test/synced_status/tikv1.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv1 jenkins 27297 17.4 0.4 3782272 1682724 ? Sl 15:16 0:04 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20161 --status-addr 127.0.0.1:20182 --log-file /tmp/tidb_cdc_test/synced_status/tikv2.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv2 jenkins 27298 24.9 0.4 3819648 1731796 ? Sl 15:16 0:06 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20162 --status-addr 127.0.0.1:20183 --log-file /tmp/tidb_cdc_test/synced_status/tikv3.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv3 jenkins 27300 24.4 0.4 3812996 1721868 ? Sl 15:16 0:06 tikv-server --pd 127.0.0.1:2479 -A 127.0.0.1:21160 --status-addr 127.0.0.1:21180 --log-file /tmp/tidb_cdc_test/synced_status/tikv_down.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv_down' ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 4972 0 --:--:-- --:--:-- --:--:-- 5062 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-30 15:16:30.108","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"2024-05-30 15:16:24.257","now_ts":"2024-05-30 15:16:31.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:16:30.108","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-05-30' '15:16:24.257","now_ts":"2024-05-30' '15:16:31.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:16:30.108","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-05-30' '15:16:24.257","now_ts":"2024-05-30' '15:16:31.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 11211 0 --:--:-- --:--:-- --:--:-- 11296 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-30 15:16:31.108","puller_resolved_ts":"2024-05-30 15:16:31.108","last_synced_ts":"2024-05-30 15:16:24.257","now_ts":"2024-05-30 15:18:41.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:16:31.108","puller_resolved_ts":"2024-05-30' '15:16:31.108","last_synced_ts":"2024-05-30' '15:16:24.257","now_ts":"2024-05-30' '15:18:41.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:16:31.108","puller_resolved_ts":"2024-05-30' '15:16:31.108","last_synced_ts":"2024-05-30' '15:16:24.257","now_ts":"2024-05-30' '15:18:41.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tidb conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f217752700013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:30346, start at 2024-05-30 15:19:03.626117343 +0800 CST m=+5.338922545 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:21:03.634 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:19:03.630 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:09:03.630 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f217752700013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:30346, start at 2024-05-30 15:19:03.626117343 +0800 CST m=+5.338922545 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:21:03.634 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:19:03.630 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:09:03.630 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f217753ec0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:30438, start at 2024-05-30 15:19:03.700291771 +0800 CST m=+5.359232350 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:21:03.708 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:19:03.675 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:09:03.675 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.31741.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='450115285543813121 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115285543813121 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=450115285543813121 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Thu May 30 15:19:10 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.3177731779.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:19:13 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/32a58343-6b0d-424e-a4ec-4687d12e8379 {"id":"32a58343-6b0d-424e-a4ec-4687d12e8379","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc85dafdfbf 32a58343-6b0d-424e-a4ec-4687d12e8379 /tidb/cdc/default/default/upstream/7374688778498715824 {"id":7374688778498715824,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/32a58343-6b0d-424e-a4ec-4687d12e8379 {"id":"32a58343-6b0d-424e-a4ec-4687d12e8379","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc85dafdfbf 32a58343-6b0d-424e-a4ec-4687d12e8379 /tidb/cdc/default/default/upstream/7374688778498715824 {"id":7374688778498715824,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/32a58343-6b0d-424e-a4ec-4687d12e8379 {"id":"32a58343-6b0d-424e-a4ec-4687d12e8379","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc85dafdfbf 32a58343-6b0d-424e-a4ec-4687d12e8379 /tidb/cdc/default/default/upstream/7374688778498715824 {"id":7374688778498715824,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=450115285543813121 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.31828.out cli changefeed create --start-ts=450115285543813121 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7374688778498715824,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T15:19:13.901160871+08:00","start_ts":450115285543813121,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115285543813121,"checkpoint_ts":450115285543813121,"checkpoint_time":"2024-05-30 15:19:08.980"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 exists + sleep 5 + kill_tidb ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 30342 4.5 0.0 2968860 162400 ? Sl 15:18 0:00 tidb-server -P 4000 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1717053538276720914.toml --store tikv --path 127.0.0.1:2379 --status=10080 --log-file /tmp/tidb_cdc_test/synced_status/tidb.log jenkins 30346 13.2 0.0 3026776 183244 ? Sl 15:18 0:02 tidb-server -P 4001 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1717053538279931347.toml --store tikv --path 127.0.0.1:2379 --status=10081 --log-file /tmp/tidb_cdc_test/synced_status/tidb_other.log jenkins 30438 13.8 0.0 2517408 211404 ? Sl 15:18 0:03 tidb-server -P 3306 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1717053538332978802.toml --store tikv --path 127.0.0.1:2479 --status=20080 --log-file /tmp/tidb_cdc_test/synced_status/tidb_down.log' ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 2920 0 --:--:-- --:--:-- --:--:-- 2927 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-30 15:19:19.331","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"2024-05-30 15:19:15.830","now_ts":"2024-05-30 15:19:20.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:19:19.331","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-05-30' '15:19:15.830","now_ts":"2024-05-30' '15:19:20.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:19:19.331","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-05-30' '15:19:15.830","now_ts":"2024-05-30' '15:19:20.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2727 0 --:--:-- --:--:-- --:--:-- 2762 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-30 15:21:29.481","puller_resolved_ts":"2024-05-30 15:21:25.530","last_synced_ts":"2024-05-30 15:19:15.830","now_ts":"2024-05-30 15:21:31.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:21:29.481","puller_resolved_ts":"2024-05-30' '15:21:25.530","last_synced_ts":"2024-05-30' '15:19:15.830","now_ts":"2024-05-30' '15:21:31.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-30' '15:21:29.481","puller_resolved_ts":"2024-05-30' '15:21:25.530","last_synced_ts":"2024-05-30' '15:19:15.830","now_ts":"2024-05-30' '15:21:31.000","info":"Data' syncing is 'finished"}' ++ jq -r .info + info='Data syncing is finished' + target_message='Data syncing is finished' + '[' 'Data syncing is finished' '!=' 'Data syncing is finished' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_failpoint conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Starting Downstream PD... Release Version: v7.5.1-15-g634e05a87 Edition: Community Git Commit Hash: 634e05a87ee0459846e48e8acea3f6bc5f6973a7 Git Branch: release-7.5 UTC Build Time: 2024-05-28 12:26:04 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 1a2c88dcc22c62e1f138d440956214ce367e838b Git Commit Branch: release-7.5 UTC Build Time: 2024-05-28 10:03:11 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-164-g39ea2b30d3 Edition: Community Git Commit Hash: 39ea2b30d32ab8cf486f30ca318cf2e4bd99eaef Git Branch: release-7.5 UTC Build Time: 2024-05-29 15:11:39 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2181b824001b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:32935, start at 2024-05-30 15:21:53.978534943 +0800 CST m=+5.107652888 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:23:53.985 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:21:53.979 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:11:53.979 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2181b824001b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:32935, start at 2024-05-30 15:21:53.978534943 +0800 CST m=+5.107652888 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:23:53.985 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:21:53.979 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:11:53.979 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 180 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63f2181bbe40009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-7-5-pull-cdc-integration-pulsar-test-462-sdxhg, pid:32992, start at 2024-05-30 15:21:54.17817508 +0800 CST m=+5.249952109 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240530-15:23:54.185 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240530-15:21:54.169 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240530-15:11:54.169 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-26-gdb0862135 Edition: Community Git Commit Hash: db0862135d4a94904069a37ec50c69cfe6099952 Git Branch: HEAD UTC Build Time: 2024-05-27 08:18:38 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-05-27 08:24:01 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-26-gdb0862135"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["db0862135d4a94904069a37ec50c69cfe6099952"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status + export 'GO_FAILPOINTS=github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.34217.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='450115329727922177 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 450115329727922177 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=450115329727922177 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Thu May 30 15:21:58 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.3424334245.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 30 May 2024 07:22:02 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9e1f92ff-421a-41a8-b82a-76cb3d2f904a {"id":"9e1f92ff-421a-41a8-b82a-76cb3d2f904a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc860465cc1 9e1f92ff-421a-41a8-b82a-76cb3d2f904a /tidb/cdc/default/default/upstream/7374689511184418107 {"id":7374689511184418107,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9e1f92ff-421a-41a8-b82a-76cb3d2f904a {"id":"9e1f92ff-421a-41a8-b82a-76cb3d2f904a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc860465cc1 9e1f92ff-421a-41a8-b82a-76cb3d2f904a /tidb/cdc/default/default/upstream/7374689511184418107 {"id":7374689511184418107,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9e1f92ff-421a-41a8-b82a-76cb3d2f904a {"id":"9e1f92ff-421a-41a8-b82a-76cb3d2f904a","address":"127.0.0.1:8300","version":"v7.5.1-58-g8f92142d7"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318fc860465cc1 9e1f92ff-421a-41a8-b82a-76cb3d2f904a /tidb/cdc/default/default/upstream/7374689511184418107 {"id":7374689511184418107,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=450115329727922177 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.34288.out cli changefeed create --start-ts=450115329727922177 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7374689511184418107,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-30T15:22:02.527744162+08:00","start_ts":450115329727922177,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-58-g8f92142d7","resolved_ts":450115329727922177,"checkpoint_ts":450115329727922177,"checkpoint_time":"2024-05-30 15:21:57.529"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x + sleep 20 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 8247 0 --:--:-- --:--:-- --:--:-- 8310 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-30 15:21:57.529","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-05-30 15:22:24.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:21:57.529","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-30' '15:22:24.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-30' '15:21:57.529","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-30' '15:22:24.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + export GO_FAILPOINTS= + GO_FAILPOINTS= + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + check_logs /tmp/tidb_cdc_test/synced_status ++ date + echo '[Thu May 30 15:22:34 CST 2024] <<<<<< run test case synced_status success! >>>>>>' [Thu May 30 15:22:34 CST 2024] <<<<<< run test case synced_status success! >>>>>> + stop_tidb_cluster \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_pulsar_test-462/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // parallel [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] End of Pipeline Finished: SUCCESS