Started by user Jenkins Admin Obtained pipelines/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test.groovy from git https://github.com/PingCAP-QE/ci.git Loading library tipipeline@main Library tipipeline@main is cached. Copying from home. [Pipeline] Start of Pipeline [Pipeline] readJSON [Pipeline] readTrusted Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-1tpwq Agent p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-1tpwq is provisioned from template p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-204nm-bxpxb --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "ae7364d888818fec0981cbb29f9e1aa53223585f" jenkins/label: "p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-204nm" name: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-1tpwq" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-1tpwq" - name: "JENKINS_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-1tpwq" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-1tpwq in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout The recommended git tool is: git No credentials specified Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Commit message: "fix(br): use failpoint tidb-server instead (#2951)" First time build. Skipping changelog. [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] timeout Timeout set to expire in 1 hr 0 min [Pipeline] { [Pipeline] stage [Pipeline] { (Debug info) [Pipeline] sh > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 > git rev-list --no-walk a0a6bb5614d55afbab88f0d1ae459423175a5407 # timeout=10 + printenv PROW_JOB_ID=8fa694ce-ccab-4faa-bb8b-f95ef2091797 JENKINS_NODE_COOKIE=c6e071c6-a4a9-492f-8bd7-86281b681a7e BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/ GOLANG_VERSION=1.21.6 HOSTNAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-1tpwq HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Debug info BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test","buildid":"1787445299229233156","prowjobid":"8fa694ce-ccab-4faa-bb8b-f95ef2091797","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct _=/usr/bin/printenv POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#61 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233156 GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=tests SHLVL=3 HOME=/home/jenkins POD_LABEL=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-204nm GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-96975af6684579e8d6d260237b9d07083cbcbab738d03b878fda5c70d2407e60 NODE_LABELS=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-204nm p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-1tpwq GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-1tpwq GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=61 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz + echo ------------------------- ------------------------- + go env GO111MODULE='' GOARCH='amd64' GOBIN='' GOCACHE='/home/jenkins/.cache/go-build' GOENV='/home/jenkins/.config/go/env' GOEXE='' GOEXPERIMENT='' GOFLAGS='' GOHOSTARCH='amd64' GOHOSTOS='linux' GOINSECURE='' GOMODCACHE='/go/pkg/mod' GONOPROXY='' GONOSUMDB='' GOOS='linux' GOPATH='/go' GOPRIVATE='' GOPROXY='http://goproxy.apps.svc,https://proxy.golang.org,direct' GOROOT='/usr/local/go' GOSUMDB='sum.golang.org' GOTMPDIR='' GOTOOLCHAIN='auto' GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64' GOVCS='' GOVERSION='go1.21.6' GCCGO='gccgo' GOAMD64='v1' AR='ar' CC='gcc' CXX='g++' CGO_ENABLED='1' GOMOD='/dev/null' GOWORK='' CGO_CFLAGS='-O2 -g' CGO_CPPFLAGS='' CGO_CXXFLAGS='-O2 -g' CGO_FFLAGS='-O2 -g' CGO_LDFLAGS='-O2 -g' PKG_CONFIG='pkg-config' GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1029644346=/tmp/go-build -gno-record-gcc-switches' + echo ------------------------- ------------------------- + echo 'debug command: kubectl -n jenkins-tiflow exec -ti p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-1tpwq bash' debug command: kubectl -n jenkins-tiflow exec -ti p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-1tpwq bash [Pipeline] container [Pipeline] { [Pipeline] sh + dig github.com ; <<>> DiG 9.18.16 <<>> github.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59074 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ; COOKIE: 12de596d8937a4fd (echoed) ;; QUESTION SECTION: ;github.com. IN A ;; ANSWER SECTION: github.com. 26 IN A 20.205.243.166 ;; Query time: 0 msec ;; SERVER: 169.254.25.10#53(169.254.25.10) (UDP) ;; WHEN: Mon May 06 11:33:00 UTC 2024 ;; MSG SIZE rcvd: 77 [Pipeline] script [Pipeline] { [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Check diff files) [Pipeline] container [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $token [Pipeline] { [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/11026/files?page=1&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/11026/files?page=1&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/11026/files?page=2&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/11026/files?page=2&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] } [Pipeline] // withCredentials [Pipeline] echo pr_diff_files: [examples/java/pom.xml] [Pipeline] echo diff file not matched: examples/java/pom.xml [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Checkout) [Pipeline] timeout Timeout set to expire in 10 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] cache Cache restored successfully (git/pingcap/tiflow/rev-936b4d4) 203908608 bytes in 3.29 secs (62008984 bytes/sec) [Pipeline] { [Pipeline] retry [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] sh git version 2.36.6 Reinitialized existing Git repository in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/.git/ .git HEAD is now at 936b4d4d2 kafka(ticdc): Fix java example license issue (#11024) POST git-upload-pack (686 bytes) POST git-upload-pack (1023 bytes) POST git-upload-pack (gzip 2173 to 1144 bytes) From https://github.com/pingcap/tiflow 3582359ab..38eb04081 release-8.1 -> origin/release-8.1 * [new ref] refs/pull/11026/head -> origin/pr/11026/head Previous HEAD position was 936b4d4d2 kafka(ticdc): Fix java example license issue (#11024) HEAD is now at 38eb04081 *(ticdc): Initialize changefeed/processor asynchronously (#10832) (#10985) ๐Ÿšง Checkouting to base SHA:38eb040812f817df410abe870952629df60fd01f... HEAD is now at 38eb04081 *(ticdc): Initialize changefeed/processor asynchronously (#10832) (#10985) โœ… Checked. ๐ŸŽ‰ ๐Ÿงพ HEAD info: 38eb040812f817df410abe870952629df60fd01f 38eb04081 *(ticdc): Initialize changefeed/processor asynchronously (#10832) (#10985) c78adb6f1 codec(ticdc): add more logs to the avro encoding to help detect potential key encoding failed issure (#10901) (#11002) 76cd91300 cli(ticdc): allow client authentication to be enabled without tls (#11005) (#11016) ๐Ÿšง Pre-merge heads of pull requests to base SHA: 38eb040812f817df410abe870952629df60fd01f ... Updating 38eb04081..e1a27adc6 Fast-forward examples/java/pom.xml | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) ๐Ÿงพ Pre-merged result: e1a27adc69acc9d50db0c3fea2945bbd6d983e30 e1a27adc6 fix java example license issue 38eb04081 *(ticdc): Initialize changefeed/processor asynchronously (#10832) (#10985) c78adb6f1 codec(ticdc): add more logs to the avro encoding to help detect potential key encoding failed issure (#10901) (#11002) โœ… Pre merged ๐ŸŽ‰ โœ… ~~~~~All done.~~~~~~ [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // retry [Pipeline] } Cache not saved (git/pingcap/tiflow/rev-38eb040-e1a27ad already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (prepare) [Pipeline] timeout Timeout set to expire in 20 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/third_party_download [Pipeline] { [Pipeline] retry [Pipeline] { [Pipeline] sh + cd ../tiflow + ./scripts/download-integration-test-binaries.sh release-8.1 Download binaries... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 41 100 41 0 0 472 0 --:--:-- --:--:-- --:--:-- 476 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 41 100 41 0 0 1330 0 --:--:-- --:--:-- --:--:-- 1366 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 41 100 41 0 0 1566 0 --:--:-- --:--:-- --:--:-- 1576 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 41 100 41 0 0 868 0 --:--:-- --:--:-- --:--:-- 872 >>> download tidb-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tidb/c8e6662d10deed030ef864169ae27718f5a61991/centos7/tidb-server.tar.gz 2024-05-06 19:33:27 URL:http://fileserver.pingcap.net/download/builds/pingcap/tidb/c8e6662d10deed030ef864169ae27718f5a61991/centos7/tidb-server.tar.gz [526223028/526223028] -> "tmp/tidb-server.tar.gz" [1] >>> download pd-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/pd/13eaf8aaa661dc0d3885ac9e59145f69752cee2d/centos7/pd-server.tar.gz 2024-05-06 19:33:44 URL:http://fileserver.pingcap.net/download/builds/pingcap/pd/13eaf8aaa661dc0d3885ac9e59145f69752cee2d/centos7/pd-server.tar.gz [184187328/184187328] -> "tmp/pd-server.tar.gz" [1] >>> download tikv-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tikv/56613f7c3e28c02853cc51d15bc1b77f68b58be8/centos7/tikv-server.tar.gz 2024-05-06 19:34:24 URL:http://fileserver.pingcap.net/download/builds/pingcap/tikv/56613f7c3e28c02853cc51d15bc1b77f68b58be8/centos7/tikv-server.tar.gz [834867730/834867730] -> "tmp/tikv-server.tar.gz" [1] >>> download tiflash.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tiflash/release-8.1/eb96e720183ea798458afc99cfb034864ed75c4c/centos7/tiflash.tar.gz 2024-05-06 19:34:55 URL:http://fileserver.pingcap.net/download/builds/pingcap/tiflash/release-8.1/eb96e720183ea798458afc99cfb034864ed75c4c/centos7/tiflash.tar.gz [455317481/455317481] -> "tmp/tiflash.tar.gz" [1] >>> download minio.tar.gz from http://fileserver.pingcap.net/download/minio.tar.gz 2024-05-06 19:35:01 URL:http://fileserver.pingcap.net/download/minio.tar.gz [17718777/17718777] -> "tmp/minio.tar.gz" [1] >>> download go-ycsb from http://fileserver.pingcap.net/download/builds/pingcap/go-ycsb/test-br/go-ycsb 2024-05-06 19:35:04 URL:http://fileserver.pingcap.net/download/builds/pingcap/go-ycsb/test-br/go-ycsb [45975512/45975512] -> "third_bin/go-ycsb" [1] >>> download jq from http://fileserver.pingcap.net/download/builds/pingcap/test/jq-1.6/jq-linux64 2024-05-06 19:35:04 URL:http://fileserver.pingcap.net/download/builds/pingcap/test/jq-1.6/jq-linux64 [3953824/3953824] -> "third_bin/jq" [1] >>> download etcd.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/etcd-v3.4.7-linux-amd64.tar.gz 2024-05-06 19:35:05 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/etcd-v3.4.7-linux-amd64.tar.gz [17310840/17310840] -> "tmp/etcd.tar.gz" [1] >>> download sync_diff_inspector.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/sync_diff_inspector_hash-d671b084_linux-amd64.tar.gz 2024-05-06 19:35:10 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/sync_diff_inspector_hash-d671b084_linux-amd64.tar.gz [79877126/79877126] -> "tmp/sync_diff_inspector.tar.gz" [1] >>> download schema-registry.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/schema-registry.tar.gz 2024-05-06 19:35:23 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/schema-registry.tar.gz [278386006/278386006] -> "tmp/schema-registry.tar.gz" [1] Download SUCCESS + ls -alh ./bin total 1.9G drwxr-sr-x 6 jenkins jenkins 4.0K May 6 19:35 . drwxr-sr-x 19 jenkins jenkins 4.0K May 6 19:35 .. drwxr-sr-x 2 jenkins jenkins 4.0K May 19 2023 bin drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 etc -rwxr-xr-x 1 jenkins jenkins 17M Apr 2 2020 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M May 6 19:35 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M May 6 19:35 jq drwxr-sr-x 3 jenkins jenkins 4.0K May 10 2023 lib lrwxrwxrwx 1 jenkins jenkins 13 May 5 14:57 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1016K Nov 7 01:00 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 May 5 14:57 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 358K Nov 7 01:00 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 May 5 14:57 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 May 5 14:57 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.6M May 5 14:18 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 272M May 5 14:58 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M Jul 29 2020 minio -rwxr-xr-x 1 jenkins jenkins 37M Apr 30 15:24 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 44M Apr 30 15:24 pd-ctl -rwxr-xr-x 1 jenkins jenkins 36M Apr 30 15:24 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 32M Apr 30 15:24 pd-recover -rwxr-xr-x 1 jenkins jenkins 106M Apr 30 15:23 pd-server -rwxr-xr-x 1 jenkins jenkins 26M Apr 30 15:24 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 32M Apr 30 15:24 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 share -rwxr-xr-x 1 jenkins jenkins 32M Apr 30 15:24 stores-dump -rwxr-xr-x 1 jenkins jenkins 192M Sep 22 2023 sync_diff_inspector -rwxr-xr-x 1 jenkins jenkins 205M May 6 18:40 tidb-server -rwxr-xr-x 1 jenkins jenkins 379M May 5 14:58 tiflash -rwxr-xr-x 1 jenkins jenkins 367M Apr 30 15:02 tikv-server + make check_third_party_binary /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/tidb-server /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/tikv-server /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/pd-server /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/tiflash /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/pd-ctl /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/sync_diff_inspector /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/go-ycsb /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/etcdctl /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/jq /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/minio /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/bin/schema-registry-start + cd - /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/third_party_download + mkdir -p bin + mv ../tiflow/bin/bin ../tiflow/bin/etc ../tiflow/bin/etcdctl ../tiflow/bin/go-ycsb ../tiflow/bin/jq ../tiflow/bin/lib ../tiflow/bin/libc++.so.1 ../tiflow/bin/libc++.so.1.0 ../tiflow/bin/libc++abi.so.1 ../tiflow/bin/libc++abi.so.1.0 ../tiflow/bin/libgmssl.so ../tiflow/bin/libgmssl.so.3 ../tiflow/bin/libgmssl.so.3.0 ../tiflow/bin/libtiflash_proxy.so ../tiflow/bin/minio ../tiflow/bin/pd-api-bench ../tiflow/bin/pd-ctl ../tiflow/bin/pd-heartbeat-bench ../tiflow/bin/pd-recover ../tiflow/bin/pd-server ../tiflow/bin/pd-tso-bench ../tiflow/bin/regions-dump ../tiflow/bin/share ../tiflow/bin/stores-dump ../tiflow/bin/sync_diff_inspector ../tiflow/bin/tidb-server ../tiflow/bin/tiflash ../tiflow/bin/tikv-server ./bin/ + ls -alh ./bin total 1.9G drwxr-sr-x 6 jenkins jenkins 4.0K May 6 19:35 . drwxr-sr-x 3 jenkins jenkins 4.0K May 6 19:35 .. drwxr-sr-x 2 jenkins jenkins 4.0K May 19 2023 bin drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 etc -rwxr-xr-x 1 jenkins jenkins 17M Apr 2 2020 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M May 6 19:35 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M May 6 19:35 jq drwxr-sr-x 3 jenkins jenkins 4.0K May 10 2023 lib lrwxrwxrwx 1 jenkins jenkins 13 May 5 14:57 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1016K Nov 7 01:00 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 May 5 14:57 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 358K Nov 7 01:00 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 May 5 14:57 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 May 5 14:57 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.6M May 5 14:18 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 272M May 5 14:58 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M Jul 29 2020 minio -rwxr-xr-x 1 jenkins jenkins 37M Apr 30 15:24 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 44M Apr 30 15:24 pd-ctl -rwxr-xr-x 1 jenkins jenkins 36M Apr 30 15:24 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 32M Apr 30 15:24 pd-recover -rwxr-xr-x 1 jenkins jenkins 106M Apr 30 15:23 pd-server -rwxr-xr-x 1 jenkins jenkins 26M Apr 30 15:24 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 32M Apr 30 15:24 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 share -rwxr-xr-x 1 jenkins jenkins 32M Apr 30 15:24 stores-dump -rwxr-xr-x 1 jenkins jenkins 192M Sep 22 2023 sync_diff_inspector -rwxr-xr-x 1 jenkins jenkins 205M May 6 18:40 tidb-server -rwxr-xr-x 1 jenkins jenkins 379M May 5 14:58 tiflash -rwxr-xr-x 1 jenkins jenkins 367M Apr 30 15:02 tikv-server + ./bin/tidb-server -V Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore + ./bin/pd-server -V Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 + ./bin/tikv-server -V TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + ./bin/tiflash --version TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored + ./bin/sync_diff_inspector --version App Name: sync_diff_inspector v2.0 Release Version: v7.4.0 Git Commit Hash: d671b0840063bc2532941f02e02e12627402844c Git Branch: heads/refs/tags/v7.4.0 UTC Build Time: 2023-09-22 03:51:56 Go Version: go1.21.1 [Pipeline] } [Pipeline] // retry [Pipeline] } [Pipeline] // dir [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] cache Cache not restored (no such key found) [Pipeline] { [Pipeline] sh + ls -alh ./bin total 8.0K drwxr-sr-x 2 jenkins jenkins 4.0K May 6 19:35 . drwxr-sr-x 19 jenkins jenkins 4.0K May 6 19:35 .. + '[' -f ./bin/cdc ']' + make cdc CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v8.1.0-alpha-23-ge1a27adc6" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-06 11:35:27" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e1a27adc69acc9d50db0c3fea2945bbd6d983e30" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v8.1.0-alpha-23-ge1a27adc6"' -o bin/cdc ./cmd/cdc go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading github.com/pingcap/failpoint v0.0.0-20220801062533-2eaa32854a6c go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20240415145106-cd9c676e9ba4 go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/pingcap/kvproto v0.0.0-20240227073058-929ab83f9754 go: downloading github.com/tikv/pd/client v0.0.0-20240322051414-fb9e2d561b6e go: downloading github.com/coreos/go-semver v0.3.1 go: downloading golang.org/x/sync v0.7.0 go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/BurntSushi/toml v1.3.2 go: downloading github.com/fatih/color v1.16.0 go: downloading github.com/prometheus/client_golang v1.19.0 go: downloading golang.org/x/net v0.24.0 go: downloading github.com/spf13/cobra v1.8.0 go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20240410110152-5fc42c9be2f5 go: downloading google.golang.org/grpc v1.62.1 go: downloading github.com/IBM/sarama v1.41.2 go: downloading github.com/gin-gonic/gin v1.9.1 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20240409022718-714958ccd4d5 go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/apache/pulsar-client-go v0.11.0 go: downloading github.com/aws/aws-sdk-go-v2 v1.19.1 go: downloading github.com/pingcap/tidb-tools v0.0.0-20240305021104-9f9bea84490b go: downloading github.com/imdario/mergo v0.3.16 go: downloading github.com/DATA-DOG/go-sqlmock v1.5.0 go: downloading github.com/xdg/scram v1.0.5 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/google/btree v1.1.2 go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading github.com/google/uuid v1.6.0 go: downloading go.uber.org/atomic v1.11.0 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading go.etcd.io/etcd/server/v3 v3.5.12 go: downloading golang.org/x/time v0.5.0 go: downloading github.com/swaggo/files v0.0.0-20210815190702-a29dd2bc99b2 go: downloading github.com/swaggo/gin-swagger v1.2.0 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.1 go: downloading github.com/KimMachineGun/automemlimit v0.2.4 go: downloading github.com/aws/aws-sdk-go v1.50.0 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/shirou/gopsutil/v3 v3.24.2 go: downloading github.com/r3labs/diff v1.1.0 go: downloading github.com/soheilhy/cmux v0.1.5 go: downloading github.com/stretchr/testify v1.9.0 go: downloading cloud.google.com/go v0.112.2 go: downloading github.com/tinylib/msgp v1.1.6 go: downloading github.com/cenkalti/backoff/v4 v4.2.1 go: downloading github.com/YangKeao/seahash v0.0.0-20240229041150-e7bf269c3140 go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading github.com/klauspost/compress v1.17.8 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/benbjohnson/clock v1.3.5 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/goccy/go-json v0.10.2 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/hashicorp/golang-lru v0.5.1 go: downloading github.com/uber-go/atomic v1.4.0 go: downloading go.etcd.io/etcd/pkg/v3 v3.5.12 go: downloading golang.org/x/sys v0.19.0 go: downloading github.com/jcmturner/gokrb5/v8 v8.4.4 go: downloading github.com/segmentio/kafka-go v0.4.41-0.20230526171612-f057b1d369cd go: downloading github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 go: downloading golang.org/x/oauth2 v0.18.0 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading github.com/gavv/monotime v0.0.0-20190418164738-30dba4353424 go: downloading github.com/go-mysql-org/go-mysql v1.7.1-0.20240314115043-2199dfb0ba98 go: downloading github.com/pingcap/check v0.0.0-20211026125417-57bd13f7b5f0 go: downloading github.com/pingcap/tidb-dashboard v0.0.0-20240326110213-9768844ff5d7 go: downloading github.com/xdg/stringprep v1.0.3 go: downloading golang.org/x/crypto v0.22.0 go: downloading github.com/swaggo/swag v1.16.3 go: downloading github.com/golang/mock v1.6.0 go: downloading github.com/containerd/cgroups v1.0.4 go: downloading github.com/go-playground/validator/v10 v10.14.0 go: downloading github.com/pelletier/go-toml/v2 v2.0.8 go: downloading github.com/gin-contrib/sse v0.1.0 go: downloading github.com/ugorji/go/codec v1.2.11 go: downloading google.golang.org/protobuf v1.33.0 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/prometheus/common v0.52.2 go: downloading github.com/prometheus/procfs v0.13.0 go: downloading gorm.io/gorm v1.24.5 go: downloading golang.org/x/term v0.19.0 go: downloading github.com/aws/smithy-go v1.13.5 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/golang/protobuf v1.5.4 go: downloading github.com/bits-and-blooms/bitset v1.4.0 go: downloading github.com/linkedin/goavro/v2 v2.11.1 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/eapache/go-resiliency v1.4.0 go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 go: downloading github.com/eapache/queue v1.1.0 go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/jcmturner/gofork v1.7.6 go: downloading github.com/philhofer/fwd v1.1.1 go: downloading cloud.google.com/go/compute/metadata v0.2.3 go: downloading cloud.google.com/go/iam v1.1.7 go: downloading cloud.google.com/go/compute v1.25.1 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading google.golang.org/api v0.170.0 go: downloading github.com/pierrec/lz4 v2.6.1+incompatible go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 go: downloading go.etcd.io/bbolt v1.3.9 go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.22.0 go: downloading go.opentelemetry.io/otel/sdk v1.22.0 go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading github.com/stretchr/objx v0.5.2 go: downloading github.com/cakturk/go-netstat v0.0.0-20200220111822-e5b49efee7a5 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240401170217-c3f982113cda go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading github.com/AthenZ/athenz v1.10.39 go: downloading github.com/jcmturner/dnsutils/v2 v2.0.0 go: downloading github.com/hashicorp/go-uuid v1.0.3 go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/Masterminds/semver v1.5.0 go: downloading github.com/siddontang/go-log v0.0.0-20180807004314-8d05993dda07 go: downloading github.com/siddontang/go v0.0.0-20180604090527-bdc77568d726 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/xdg-go/scram v1.1.2 go: downloading golang.org/x/text v0.14.0 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 go: downloading golang.org/x/exp v0.0.0-20240409090435-93d18d7e34b8 go: downloading github.com/joomcode/errorx v1.0.1 go: downloading github.com/KyleBanks/depth v1.2.1 go: downloading github.com/go-openapi/spec v0.21.0 go: downloading golang.org/x/tools v0.20.0 go: downloading github.com/godbus/dbus/v5 v5.0.4 go: downloading github.com/opencontainers/runtime-spec v1.0.2 go: downloading github.com/cilium/ebpf v0.4.0 go: downloading github.com/gabriel-vasile/mimetype v1.4.2 go: downloading github.com/go-playground/universal-translator v0.18.1 go: downloading github.com/leodido/go-urn v1.2.4 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 go: downloading github.com/glebarez/sqlite v1.7.0 go: downloading gorm.io/driver/mysql v1.3.3 go: downloading github.com/jinzhu/now v1.1.5 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.1 go: downloading github.com/spaolacci/murmur3 v1.1.0 go: downloading golang.org/x/mod v0.17.0 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/hashicorp/errwrap v1.0.0 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/pingcap/tipb v0.0.0-20240318032315-55a7867ddd50 go: downloading go.etcd.io/etcd/raft/v3 v3.5.12 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/jonboulle/clockwork v0.4.0 go: downloading github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 go: downloading github.com/otiai10/copy v1.2.0 go: downloading github.com/gorilla/mux v1.8.0 go: downloading github.com/blacktear23/go-proxyprotocol v1.0.6 go: downloading github.com/golang-jwt/jwt v3.2.2+incompatible go: downloading github.com/99designs/keyring v1.2.1 go: downloading github.com/pingcap/fn v1.0.0 go: downloading github.com/scalalang2/golang-fifo v0.1.5 go: downloading github.com/tidwall/btree v1.7.0 go: downloading github.com/jcmturner/rpc/v2 v2.0.3 go: downloading github.com/jcmturner/aescts/v2 v2.0.0 go: downloading github.com/tiancaiamao/appdash v0.0.0-20181126055449-889f96f722a2 go: downloading github.com/sourcegraph/appdash-data v0.0.0-20151005221446-73f23eafcf67 go: downloading github.com/mailru/easyjson v0.7.7 go: downloading github.com/aws/aws-sdk-go-v2/config v1.18.30 go: downloading github.com/aws/aws-sdk-go-v2/service/glue v1.58.1 go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.13.29 go: downloading github.com/jarcoal/httpmock v1.2.0 go: downloading github.com/shopspring/decimal v1.3.0 go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/gorilla/websocket v1.5.1 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading github.com/glebarez/go-sqlite v1.21.2 go: downloading github.com/xdg-go/pbkdf2 v1.0.0 go: downloading github.com/xdg-go/stringprep v1.0.4 go: downloading github.com/go-ozzo/ozzo-validation/v4 v4.3.0 go: downloading github.com/jinzhu/inflection v1.0.0 go: downloading github.com/go-playground/locales v0.14.1 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/go-openapi/jsonpointer v0.21.0 go: downloading github.com/go-openapi/jsonreference v0.21.0 go: downloading github.com/go-openapi/swag v0.23.0 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.1 go: downloading github.com/go-logr/logr v1.4.1 go: downloading go.opentelemetry.io/proto/otlp v1.1.0 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/edwingeng/deque v0.0.0-20191220032131-8596380dee17 go: downloading github.com/golang-jwt/jwt/v4 v4.5.0 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading go.uber.org/mock v0.4.0 go: downloading go.etcd.io/etcd/client/v2 v2.305.12 go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading github.com/danjacques/gofslock v0.0.0-20240212154529-d899e02bfe22 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/dvsekhvalnov/jose2go v1.5.0 go: downloading github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c go: downloading github.com/mtibben/percent v0.2.1 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.0 go: downloading github.com/google/pprof v0.0.0-20240117000934-35fc243c5815 go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.6 go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.3.37 go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.12.14 go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.14 go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.20.1 go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 go: downloading modernc.org/libc v1.37.1 go: downloading modernc.org/sqlite v1.27.0 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.36 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/jfcg/sixb v1.3.8 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/vbauerster/mpb/v7 v7.5.3 go: downloading github.com/pingcap/badger v1.5.1-0.20230103063557-828f39b09b6d go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/mattn/go-runewidth v0.0.15 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.30 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.12.0 go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.30 go: downloading github.com/josharian/intern v1.0.0 go: downloading github.com/golang-jwt/jwt/v5 v5.2.0 go: downloading github.com/golang/glog v1.2.0 go: downloading k8s.io/api v0.28.6 go: downloading github.com/emirpasic/gods v1.18.1 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/ncw/directio v1.0.5 go: downloading k8s.io/apimachinery v0.28.6 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading github.com/google/gofuzz v1.2.0 go: downloading k8s.io/klog/v2 v2.120.1 go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd go: downloading github.com/ardielle/ardielle-go v1.5.2 go: downloading github.com/jmespath/go-jmespath v0.4.0 go: downloading modernc.org/memory v1.7.2 go: downloading modernc.org/mathutil v1.6.0 go: downloading github.com/google/s2a-go v0.1.7 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading github.com/felixge/httpsnoop v1.0.4 + '[' -f ./bin/cdc_kafka_consumer ']' + make kafka_consumer CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v8.1.0-alpha-23-ge1a27adc6" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-06 11:38:23" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e1a27adc69acc9d50db0c3fea2945bbd6d983e30" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v8.1.0-alpha-23-ge1a27adc6"' -o bin/cdc_kafka_consumer ./cmd/kafka-consumer/main.go + '[' -f ./bin/cdc_storage_consumer ']' + make storage_consumer CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v8.1.0-alpha-23-ge1a27adc6" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-06 11:38:30" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e1a27adc69acc9d50db0c3fea2945bbd6d983e30" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v8.1.0-alpha-23-ge1a27adc6"' -o bin/cdc_storage_consumer ./cmd/storage-consumer/main.go + '[' -f ./bin/cdc.test ']' + make integration_test_build cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/failpoint-ctl github.com/pingcap/failpoint/failpoint-ctl go: downloading github.com/pingcap/failpoint v0.0.0-20210316064728-7acb0f0a3dfd go: downloading github.com/sergi/go-diff v1.1.0 CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v8.1.0-alpha-23-ge1a27adc6" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-06 11:38:37" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e1a27adc69acc9d50db0c3fea2945bbd6d983e30" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v8.1.0-alpha-23-ge1a27adc6"' -o bin/cdc_storage_consumer ./cmd/storage-consumer/main.go CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v8.1.0-alpha-23-ge1a27adc6" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-06 11:38:37" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e1a27adc69acc9d50db0c3fea2945bbd6d983e30" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v8.1.0-alpha-23-ge1a27adc6"' -o bin/cdc_kafka_consumer ./cmd/kafka-consumer/main.go CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v8.1.0-alpha-23-ge1a27adc6" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-06 11:38:37" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e1a27adc69acc9d50db0c3fea2945bbd6d983e30" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v8.1.0-alpha-23-ge1a27adc6"' -o bin/cdc_pulsar_consumer ./cmd/pulsar-consumer/main.go CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v8.1.0-alpha-23-ge1a27adc6" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-06 11:38:37" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e1a27adc69acc9d50db0c3fea2945bbd6d983e30" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v8.1.0-alpha-23-ge1a27adc6"' -o bin/oauth2-server ./cmd/oauth2-server/main.go go: downloading github.com/go-oauth2/oauth2/v4 v4.5.2 go: downloading github.com/tidwall/buntdb v1.3.0 go: downloading github.com/tidwall/rtred v0.1.2 go: downloading github.com/tidwall/grect v0.1.4 go: downloading github.com/tidwall/gjson v1.14.3 go: downloading github.com/tidwall/match v1.1.1 go: downloading github.com/tidwall/tinyqueue v0.1.1 go: downloading github.com/tidwall/pretty v1.2.0 $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl enable >/dev/null) go: downloading github.com/PingCAP-QE/go-sqlsmith v0.0.0-20231213065948-336e064b488d go: downloading github.com/chzyer/readline v1.5.1 go: downloading github.com/DataDog/zstd v1.5.5 go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 go: downloading github.com/deepmap/oapi-codegen v1.9.0 go: downloading github.com/gogo/gateway v1.1.0 go: downloading github.com/getkin/kin-openapi v0.80.0 go: downloading github.com/shurcooL/httpgzip v0.0.0-20190720172056-320755c1c1b0 go: downloading github.com/syndtr/goleveldb v1.0.1-0.20210305035536-64b5b1c73954 go: downloading github.com/mattn/go-shellwords v1.0.12 go: downloading github.com/ngaut/log v0.0.0-20210830112240-0124ec040aeb go: downloading go.uber.org/dig v1.13.0 go: downloading go.uber.org/ratelimit v0.2.0 go: downloading github.com/VividCortex/mysqlerr v1.0.0 go: downloading go.uber.org/goleak v1.3.0 go: downloading github.com/bradleyjkemp/grpc-tools v0.2.5 go: downloading github.com/integralist/go-findroot v0.0.0-20160518114804-ac90681525dc go: downloading github.com/jmoiron/sqlx v1.3.3 go: downloading upper.io/db.v3 v3.7.1+incompatible go: downloading github.com/andres-erbsen/clock v0.0.0-20160526145045-9e14626cd129 go: downloading github.com/improbable-eng/grpc-web v0.12.0 go: downloading github.com/ghodss/yaml v1.0.0 go: downloading github.com/rs/cors v1.7.0 go: downloading github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f CGO_ENABLED=1 GO111MODULE=on go test -p 3 --race --tags=intest -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v8.1.0-alpha-23-ge1a27adc6" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-06 11:38:37" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e1a27adc69acc9d50db0c3fea2945bbd6d983e30" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v8.1.0-alpha-23-ge1a27adc6"' -c -cover -covermode=atomic \ -coverpkg=github.com/pingcap/tiflow/... \ -o bin/cdc.test github.com/pingcap/tiflow/cmd/cdc \ || { $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null); exit 1; } CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v8.1.0-alpha-23-ge1a27adc6" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-06 11:38:37" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e1a27adc69acc9d50db0c3fea2945bbd6d983e30" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v8.1.0-alpha-23-ge1a27adc6"' -o bin/cdc ./cmd/cdc/main.go \ || { $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null); exit 1; } $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null) + ls -alh ./bin total 1.2G drwxr-sr-x 2 jenkins jenkins 4.0K May 6 19:43 . drwxr-sr-x 19 jenkins jenkins 4.0K May 6 19:35 .. -rwxr-xr-x 1 jenkins jenkins 220M May 6 19:43 cdc -rwxr-xr-x 1 jenkins jenkins 363M May 6 19:43 cdc.test -rwxr-xr-x 1 jenkins jenkins 183M May 6 19:38 cdc_kafka_consumer -rwxr-xr-x 1 jenkins jenkins 182M May 6 19:39 cdc_pulsar_consumer -rwxr-xr-x 1 jenkins jenkins 182M May 6 19:38 cdc_storage_consumer -rwxr-xr-x 1 jenkins jenkins 12M May 6 19:39 oauth2-server + ./bin/cdc version Release Version: v8.1.0-alpha-23-ge1a27adc6 Git Commit Hash: e1a27adc69acc9d50db0c3fea2945bbd6d983e30 Git Branch: HEAD UTC Build Time: 2024-05-06 11:38:37 Go Version: go version go1.21.6 linux/amd64 Failpoint Build: true [Pipeline] } Cache saved successfully (binary/pingcap/tiflow/cdc-integration-test/rev-38eb040-e1a27ad) 1194940928 bytes in 36.34 secs (32882692 bytes/sec) [Pipeline] // cache [Pipeline] cache Cache not restored (no such key found) [Pipeline] { [Pipeline] sh + cp -r ../third_party_download/bin/bin ../third_party_download/bin/etc ../third_party_download/bin/etcdctl ../third_party_download/bin/go-ycsb ../third_party_download/bin/jq ../third_party_download/bin/lib ../third_party_download/bin/libc++.so.1 ../third_party_download/bin/libc++.so.1.0 ../third_party_download/bin/libc++abi.so.1 ../third_party_download/bin/libc++abi.so.1.0 ../third_party_download/bin/libgmssl.so ../third_party_download/bin/libgmssl.so.3 ../third_party_download/bin/libgmssl.so.3.0 ../third_party_download/bin/libtiflash_proxy.so ../third_party_download/bin/minio ../third_party_download/bin/pd-api-bench ../third_party_download/bin/pd-ctl ../third_party_download/bin/pd-heartbeat-bench ../third_party_download/bin/pd-recover ../third_party_download/bin/pd-server ../third_party_download/bin/pd-tso-bench ../third_party_download/bin/regions-dump ../third_party_download/bin/share ../third_party_download/bin/stores-dump ../third_party_download/bin/sync_diff_inspector ../third_party_download/bin/tidb-server ../third_party_download/bin/tiflash ../third_party_download/bin/tikv-server ./bin/ + ls -alh ./bin total 3.0G drwxr-sr-x 6 jenkins jenkins 4.0K May 6 19:44 . drwxr-sr-x 19 jenkins jenkins 4.0K May 6 19:35 .. drwxr-sr-x 2 jenkins jenkins 4.0K May 6 19:44 bin -rwxr-xr-x 1 jenkins jenkins 220M May 6 19:43 cdc -rwxr-xr-x 1 jenkins jenkins 363M May 6 19:43 cdc.test -rwxr-xr-x 1 jenkins jenkins 183M May 6 19:38 cdc_kafka_consumer -rwxr-xr-x 1 jenkins jenkins 182M May 6 19:39 cdc_pulsar_consumer -rwxr-xr-x 1 jenkins jenkins 182M May 6 19:38 cdc_storage_consumer drwxr-sr-x 4 jenkins jenkins 4.0K May 6 19:44 etc -rwxr-xr-x 1 jenkins jenkins 17M May 6 19:44 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M May 6 19:44 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M May 6 19:44 jq drwxr-sr-x 3 jenkins jenkins 4.0K May 6 19:44 lib lrwxrwxrwx 1 jenkins jenkins 13 May 6 19:44 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1016K May 6 19:44 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 May 6 19:44 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 358K May 6 19:44 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 May 6 19:44 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 May 6 19:44 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.6M May 6 19:44 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 272M May 6 19:44 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M May 6 19:44 minio -rwxr-xr-x 1 jenkins jenkins 12M May 6 19:39 oauth2-server -rwxr-xr-x 1 jenkins jenkins 37M May 6 19:44 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 44M May 6 19:44 pd-ctl -rwxr-xr-x 1 jenkins jenkins 36M May 6 19:44 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 32M May 6 19:44 pd-recover -rwxr-xr-x 1 jenkins jenkins 106M May 6 19:44 pd-server -rwxr-xr-x 1 jenkins jenkins 26M May 6 19:44 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 32M May 6 19:44 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K May 6 19:44 share -rwxr-xr-x 1 jenkins jenkins 32M May 6 19:44 stores-dump -rwxr-xr-x 1 jenkins jenkins 192M May 6 19:44 sync_diff_inspector -rwxr-xr-x 1 jenkins jenkins 205M May 6 19:44 tidb-server -rwxr-xr-x 1 jenkins jenkins 379M May 6 19:44 tiflash -rwxr-xr-x 1 jenkins jenkins 367M May 6 19:44 tikv-server [Pipeline] } Cache saved successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc) 3667979264 bytes in 106.59 secs (34412676 bytes/sec) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Tests) [Pipeline] parallel [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G00') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G01') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G02') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G03') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G04') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G05') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G06') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G07') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G08') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G09') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G10') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G11') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G12') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G13') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G14') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G15') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G16') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G17') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G00') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G01') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G02') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G03') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G04') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G05') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G06') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G07') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G08') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G09') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G10') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G11') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G12') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G13') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G14') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G15') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G16') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G17') [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-px7vh [Pipeline] node Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-b8w74 Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-vs88z Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-t7zm9 [Pipeline] node Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Agent p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-px7vh is provisioned from template p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-rhp25-t1dnp --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "450726341cc9e69c79c7b777d28fd0353da11d44" jenkins/label: "p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-rhp25" name: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-px7vh" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-px7vh" - name: "JENKINS_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-px7vh" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-px7vh in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test [Pipeline] { [Pipeline] checkout Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-nrrkt The recommended git tool is: git Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-245l7 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@613a96d3; decorates RemoteLauncher[hudson.remoting.Channel@7788417e:JNLP4-connect connection from 10.233.108.148/10.233.108.148:51140] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-drnnn Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Agent p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-245l7 is provisioned from template p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-x985g-jg6td --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "c5676708c2e62628a945a067e153fc0cfa9ff7f0" jenkins/label: "p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-x985g" name: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-245l7" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-245l7" - name: "JENKINS_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-245l7" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-245l7 in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test [Pipeline] { [Pipeline] checkout The recommended git tool is: git Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { Commit message: "fix(br): use failpoint tidb-server instead (#2951)" [Pipeline] node Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] stage [Pipeline] { (Test) [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] cache > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@79aa682e; decorates RemoteLauncher[hudson.remoting.Channel@26c9bd97:JNLP4-connect connection from 10.233.127.91/10.233.127.91:37058] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Agent p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-vs88z is provisioned from template p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-bd64b-35q8h --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "37ad22b76ef5460370e0846d063fcdd02bcd49e7" jenkins/label: "p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-bd64b" name: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-vs88z" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-vs88z" - name: "JENKINS_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-vs88z" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-vs88z in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-t7zm9 is provisioned from template p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-0crks-5sk17 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "b6721d6f606464e2a963984f21cf1654c57434c2" jenkins/label: "p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-0crks" name: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-t7zm9" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-t7zm9" - name: "JENKINS_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-t7zm9" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-t7zm9 in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-b8w74 is provisioned from template p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-52z2z-699qm --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "4cc1dd974e46fc4026f3d6fd7656ff51c229c538" jenkins/label: "p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-52z2z" name: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-b8w74" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-b8w74" - name: "JENKINS_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-b8w74" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-b8w74 in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l is provisioned from template p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-rs3q1-7r6zw --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "f9d018886803585aaa6f6cf5c47de1aaf7e10da3" jenkins/label: "p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-rs3q1" name: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l" - name: "JENKINS_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz is provisioned from template p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-whxv2-95s6n --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "279c9bd9fe0a9adb6a9c17631b3212d92fcef333" jenkins/label: "p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-whxv2" name: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz" - name: "JENKINS_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w9f11 Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Commit message: "fix(br): use failpoint tidb-server instead (#2951)" Agent p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-drnnn is provisioned from template p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-z2m3f-p9224 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "c195a07ec17c02c25ac937d80ef4de21bf0350c7" jenkins/label: "p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-z2m3f" name: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-drnnn" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-drnnn" - name: "JENKINS_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-drnnn" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 Running on p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-drnnn in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test Still waiting to schedule task โ€˜p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-nrrktโ€™ is offline Agent p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-nrrkt is provisioned from template p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-b3pdh-4v1d7 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "2b9021ce4679883ec83e5cf420bc095fe8a8f961" jenkins/label: "p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-b3pdh" name: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-nrrkt" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-nrrkt" - name: "JENKINS_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-nrrkt" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-nrrkt in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test Agent p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w9f11 is provisioned from template p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-0w7kh-0g9jv --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "c5d0a971dd1cd8c9bdcbd65b8b57c2383e2fae97" jenkins/label: "p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-0w7kh" name: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w9f11" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w9f11" - name: "JENKINS_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w9f11" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w9f11 in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc) 3667979264 bytes in 16.16 secs (226981711 bytes/sec) [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] { [Pipeline] sh [Pipeline] { [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] { [Pipeline] { + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G00 Run cases: bdr_mode capture_suicide_while_balance_table syncpoint hang_sink_suicide server_config_compatibility kafka_big_messages kafka_compression kafka_messages kafka_sink_error_resume mq_sink_lost_callback mq_sink_dispatcher kafka_column_selector kafka_column_selector_avro debezium lossy_ddl storage_csv_update PROW_JOB_ID=8fa694ce-ccab-4faa-bb8b-f95ef2091797 JENKINS_NODE_COOKIE=13bdf0dd-d041-4a8b-88c9-9688fa7e84ce BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/ GOLANG_VERSION=1.21.6 HOSTNAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-px7vh HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test","buildid":"1787445299229233156","prowjobid":"8fa694ce-ccab-4faa-bb8b-f95ef2091797","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#61 TEST_GROUP=G00 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233156 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-rhp25 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-81bc5d443e840859e7b2b4bd4bb8be00446de8a572b18ada90bce15670edaaa5 NODE_LABELS=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-rhp25 p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-px7vh GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-px7vh GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=61 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/bdr_mode/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:47:04 CST 2024] <<<<<< run test case bdr_mode success! >>>>>> [Pipeline] checkout The recommended git tool is: git [Pipeline] container [Pipeline] { [Pipeline] checkout [Pipeline] { The recommended git tool is: git [Pipeline] { [Pipeline] { [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-pst3l Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w7kft Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-dscp5 Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-7rfgh Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-njdk7 The recommended git tool is: git [Pipeline] checkout Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-81vjm Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rwfr3 [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] stage [Pipeline] { (Test) The recommended git tool is: git No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@409f9ca4; decorates RemoteLauncher[hudson.remoting.Channel@305561ca:JNLP4-connect connection from 10.233.106.116/10.233.106.116:56176] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] cache No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@5ce86d25; decorates RemoteLauncher[hudson.remoting.Channel@71b9fcd0:JNLP4-connect connection from 10.233.97.146/10.233.97.146:55614] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@2f6ccb96; decorates RemoteLauncher[hudson.remoting.Channel@1f192e01:JNLP4-connect connection from 10.233.88.17/10.233.88.17:54422] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@5026cebb; decorates RemoteLauncher[hudson.remoting.Channel@713692f1:JNLP4-connect connection from 10.233.86.39/10.233.86.39:38176] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@607bfd10; decorates RemoteLauncher[hudson.remoting.Channel@3319969c:JNLP4-connect connection from 10.233.97.186/10.233.97.186:48008] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@4ca98904; decorates RemoteLauncher[hudson.remoting.Channel@4ef6db54:JNLP4-connect connection from 10.233.90.43/10.233.90.43:34090] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@1d6f734f; decorates RemoteLauncher[hudson.remoting.Channel@1496028a:JNLP4-connect connection from 10.233.93.89/10.233.93.89:50058] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@349d6d89; decorates RemoteLauncher[hudson.remoting.Channel@50151a55:JNLP4-connect connection from 10.233.68.97/10.233.68.97:47628] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Agent p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-7rfgh is provisioned from template p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-tcn64-v93cn --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "e5c34b65b8683d954cdc63ade5213d374f7866db" jenkins/label: "p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-tcn64" name: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-7rfgh" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-7rfgh" - name: "JENKINS_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-7rfgh" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-7rfgh in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/capture_suicide_while_balance_table/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:47:07 CST 2024] <<<<<< run test case capture_suicide_while_balance_table success! >>>>>> > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Commit message: "fix(br): use failpoint tidb-server instead (#2951)" Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Agent p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rwfr3 is provisioned from template p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-6884q-6pnj1 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "03ca4f1683898e8f33d8a1dd6379e2ba7fdd46c6" jenkins/label: "p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-6884q" name: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rwfr3" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rwfr3" - name: "JENKINS_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rwfr3" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rwfr3 in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test Commit message: "fix(br): use failpoint tidb-server instead (#2951)" Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 Commit message: "fix(br): use failpoint tidb-server instead (#2951)" Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Commit message: "fix(br): use failpoint tidb-server instead (#2951)" > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 Commit message: "fix(br): use failpoint tidb-server instead (#2951)" Agent p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-dscp5 is provisioned from template p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-98v6t-frk6k --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "d37cfd601c16c0060b86dee3deb20e70aaf1f73b" jenkins/label: "p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-98v6t" name: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-dscp5" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-dscp5" - name: "JENKINS_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-dscp5" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-dscp5 in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test Agent p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w7kft is provisioned from template p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-0pn5r-4zh2t --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "40701a9af740a22d52d1134c63ec68818a691733" jenkins/label: "p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-0pn5r" name: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w7kft" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w7kft" - name: "JENKINS_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w7kft" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w7kft in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test Avoid second fetch > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 Commit message: "fix(br): use failpoint tidb-server instead (#2951)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Commit message: "fix(br): use failpoint tidb-server instead (#2951)" Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Agent p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-pst3l is provisioned from template p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-nwnml-vzm6x --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "02e897be8885c535e5c52d3041777d6448799910" jenkins/label: "p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-nwnml" name: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-pst3l" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-pst3l" - name: "JENKINS_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-pst3l" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-pst3l in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test Commit message: "fix(br): use failpoint tidb-server instead (#2951)" > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 Agent p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-njdk7 is provisioned from template p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-v7nm2-kczj3 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "7ca8d4960da50aa595a7ba24c4d42e6277f80ca4" jenkins/label: "p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-v7nm2" name: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-njdk7" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-njdk7" - name: "JENKINS_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-njdk7" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-njdk7 in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/syncpoint/run.sh using Sink-Type: storage... <<================= kafka downstream isn't support syncpoint record [Mon May 6 19:47:11 CST 2024] <<<<<< run test case syncpoint success! >>>>>> Created Pod: kubernetes jenkins-tiflow/p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3 Agent p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-81vjm is provisioned from template p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-r4b17-23809 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "93d90148caa2273dcca2a162ee59b3884514e0e7" jenkins/label: "p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-r4b17" name: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-81vjm" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-81vjm" - name: "JENKINS_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-81vjm" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-81vjm in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/hang_sink_suicide/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:47:15 CST 2024] <<<<<< run test case hang_sink_suicide success! >>>>>> Agent p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3 is provisioned from template p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-f0r2j-07w2c --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "9283080a888b6d022a1e505dea7a8488a7af1323" jenkins/label: "p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-f0r2j" name: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3" - name: "JENKINS_NAME" value: "p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3 in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/server_config_compatibility/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:47:18 CST 2024] <<<<<< run test case server_config_compatibility success! >>>>>> Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc) 3667979264 bytes in 11.33 secs (323724976 bytes/sec) [Pipeline] { [Pipeline] { [Pipeline] sh [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] checkout + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G05 Run cases: charset_gbk ddl_manager multi_source PROW_JOB_ID=8fa694ce-ccab-4faa-bb8b-f95ef2091797 JENKINS_NODE_COOKIE=f4119794-dbd3-46ab-9e12-5309ac698539 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/ GOLANG_VERSION=1.21.6 HOSTNAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-245l7 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test","buildid":"1787445299229233156","prowjobid":"8fa694ce-ccab-4faa-bb8b-f95ef2091797","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#61 TEST_GROUP=G05 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233156 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-x985g GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-81bc5d443e840859e7b2b4bd4bb8be00446de8a572b18ada90bce15670edaaa5 NODE_LABELS=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-245l7 p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-x985g GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-245l7 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=61 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/charset_gbk/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... [Pipeline] { The recommended git tool is: git [Pipeline] checkout [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { The recommended git tool is: git [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] checkout [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout [Pipeline] checkout The recommended git tool is: git The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git The recommended git tool is: git [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { No credentials specified [Pipeline] timeout Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@67a69946; decorates RemoteLauncher[hudson.remoting.Channel@779f0ce9:JNLP4-connect connection from 10.233.123.246/10.233.123.246:33248] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Timeout set to expire in 40 min Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@7abc2471; decorates RemoteLauncher[hudson.remoting.Channel@79dde872:JNLP4-connect connection from 10.233.126.119/10.233.126.119:37184] will be ignored (a typical symptom is the Git executable not being run inside a designated container) [Pipeline] withCredentials Cloning the remote Git repository Using shallow clone with depth 1 Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] { Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@22fbb3b8; decorates RemoteLauncher[hudson.remoting.Channel@526f0e8d:JNLP4-connect connection from 10.233.84.118/10.233.84.118:55832] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] { No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@242fed61; decorates RemoteLauncher[hudson.remoting.Channel@5d5bf42e:JNLP4-connect connection from 10.233.93.30/10.233.93.30:34970] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@335a4ea9; decorates RemoteLauncher[hudson.remoting.Channel@5a57f09c:JNLP4-connect connection from 10.233.72.196/10.233.72.196:60872] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@2481816b; decorates RemoteLauncher[hudson.remoting.Channel@62a604a6:JNLP4-connect connection from 10.233.100.236/10.233.100.236:57634] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow [Pipeline] { Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@5820ca27; decorates RemoteLauncher[hudson.remoting.Channel@4c188ef5:JNLP4-connect connection from 10.233.70.69/10.233.70.69:60274] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] cache Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test # timeout=10 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_big_messages/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:47:21 CST 2024] <<<<<< run test case kafka_big_messages success! >>>>>> No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@3f858d35; decorates RemoteLauncher[hudson.remoting.Channel@622555b3:JNLP4-connect connection from 10.233.69.233/10.233.69.233:37994] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 start tidb cluster in /tmp/tidb_cdc_test/charset_gbk Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Commit message: "fix(br): use failpoint tidb-server instead (#2951)" Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Commit message: "fix(br): use failpoint tidb-server instead (#2951)" Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Avoid second fetch Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 Commit message: "fix(br): use failpoint tidb-server instead (#2951)" Commit message: "fix(br): use failpoint tidb-server instead (#2951)" Commit message: "fix(br): use failpoint tidb-server instead (#2951)" Commit message: "fix(br): use failpoint tidb-server instead (#2951)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Commit message: "fix(br): use failpoint tidb-server instead (#2951)" Commit message: "fix(br): use failpoint tidb-server instead (#2951)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_compression/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:47:26 CST 2024] <<<<<< run test case kafka_compression success! >>>>>> Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_messages/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:47:30 CST 2024] <<<<<< run test case kafka_messages success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ec14d80009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-245l7, pid:1244, start at 2024-05-06 19:47:33.310817269 +0800 CST m=+5.168217041 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:49:33.317 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:47:33.302 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:37:33.302 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ec14d80009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-245l7, pid:1244, start at 2024-05-06 19:47:33.310817269 +0800 CST m=+5.168217041 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:49:33.317 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:47:33.302 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:37:33.302 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ec16c80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-245l7, pid:1319, start at 2024-05-06 19:47:33.452825533 +0800 CST m=+5.255647117 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:49:33.458 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:47:33.426 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:37:33.426 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/charset_gbk/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/charset_gbk/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_sink_error_resume/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:47:36 CST 2024] <<<<<< run test case kafka_sink_error_resume success! >>>>>> [Mon May 6 19:47:36 CST 2024] <<<<<< START cdc server in charset_gbk case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.charset_gbk.26382640.out server --log-file /tmp/tidb_cdc_test/charset_gbk/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/charset_gbk/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/mq_sink_lost_callback/run.sh using Sink-Type: storage... <<================= < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:47:39 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2c38983c-9115-4d74-a445-2d4334ffd251 {"id":"2c38983c-9115-4d74-a445-2d4334ffd251","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996056} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbadd262d 2c38983c-9115-4d74-a445-2d4334ffd251 /tidb/cdc/default/default/upstream/7365851929482060213 {"id":7365851929482060213,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2c38983c-9115-4d74-a445-2d4334ffd251 {"id":"2c38983c-9115-4d74-a445-2d4334ffd251","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996056} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbadd262d 2c38983c-9115-4d74-a445-2d4334ffd251 /tidb/cdc/default/default/upstream/7365851929482060213 {"id":7365851929482060213,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2c38983c-9115-4d74-a445-2d4334ffd251 {"id":"2c38983c-9115-4d74-a445-2d4334ffd251","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996056} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbadd262d 2c38983c-9115-4d74-a445-2d4334ffd251 /tidb/cdc/default/default/upstream/7365851929482060213 {"id":7365851929482060213,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: 82395035-d096-4c6e-9dae-d59d43350937 Info: {"upstream_id":7365851929482060213,"namespace":"default","id":"82395035-d096-4c6e-9dae-d59d43350937","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-06T19:47:39.887039082+08:00","start_ts":449575926249029633,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575926249029633,"checkpoint_ts":449575926249029633,"checkpoint_time":"2024-05-06 19:47:36.553"} + workdir=/tmp/tidb_cdc_test/charset_gbk + sink_uri=mysql://normal:123456@127.0.0.1:3306/ + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/charset_gbk ++ date + echo '[Mon May 6 19:47:39 CST 2024] <<<<<< START storage consumer in charset_gbk case >>>>>>' [Mon May 6 19:47:39 CST 2024] <<<<<< START storage consumer in charset_gbk case >>>>>> + cd /tmp/tidb_cdc_test/charset_gbk + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/charset_gbk + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/charset_gbk/cdc_storage_consumer.log --log-level debug --upstream-uri mysql://normal:123456@127.0.0.1:3306/ --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' [Mon May 6 19:47:39 CST 2024] <<<<<< run test case mq_sink_lost_callback success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/mq_sink_dispatcher/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:47:43 CST 2024] <<<<<< run test case mq_sink_dispatcher success! >>>>>> Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc) 3667979264 bytes in 21.49 secs (170716114 bytes/sec) [Pipeline] { [Pipeline] cache table charset_gbk_test0.t0 exists table charset_gbk_test0.t1 exists table charset_gbk_test1.t0 not exists for 1-th check, retry later table charset_gbk_test1.t0 exists table test.finish_mark not exists for 1-th check, retry later table test.finish_mark not exists for 2-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_column_selector/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:47:52 CST 2024] <<<<<< run test case kafka_column_selector success! >>>>>> table test.finish_mark not exists for 3-th check, retry later table test.finish_mark not exists for 4-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_column_selector_avro/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:47:55 CST 2024] <<<<<< run test case kafka_column_selector_avro success! >>>>>> Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc) 3667979264 bytes in 9.93 secs (369447028 bytes/sec) [Pipeline] { [Pipeline] cache table test.finish_mark not exists for 5-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/debezium/run.sh using Sink-Type: storage... <<================= find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/lossy_ddl/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table test.finish_mark exists check table exists success check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:48:01 CST 2024] <<<<<< run test case charset_gbk success! >>>>>> start tidb cluster in /tmp/tidb_cdc_test/lossy_ddl Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/ddl_manager/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/ddl_manager Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ee84d80019 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-px7vh, pid:2402, start at 2024-05-06 19:48:13.281078028 +0800 CST m=+5.242478863 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:13.288 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:13.288 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:13.288 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ee84d80019 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-px7vh, pid:2402, start at 2024-05-06 19:48:13.281078028 +0800 CST m=+5.242478863 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:13.288 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:13.288 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:13.288 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ee86680004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-px7vh, pid:2484, start at 2024-05-06 19:48:13.342218715 +0800 CST m=+5.208390479 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:13.352 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:13.338 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:13.338 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/lossy_ddl/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/lossy_ddl/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/lossy_ddl/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/lossy_ddl/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/lossy_ddl/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc) 3667979264 bytes in 18.57 secs (197523221 bytes/sec) [Pipeline] { [Pipeline] cache Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [Mon May 6 19:48:18 CST 2024] <<<<<< START cdc server in lossy_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.lossy_ddl.37523754.out server --log-file /tmp/tidb_cdc_test/lossy_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/lossy_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:48:21 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a4feded9-371a-49d9-82f3-f12c39cce0a7 {"id":"a4feded9-371a-49d9-82f3-f12c39cce0a7","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996098} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb75dc33 a4feded9-371a-49d9-82f3-f12c39cce0a7 /tidb/cdc/default/default/upstream/7365852094888221765 {"id":7365852094888221765,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a4feded9-371a-49d9-82f3-f12c39cce0a7 {"id":"a4feded9-371a-49d9-82f3-f12c39cce0a7","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996098} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb75dc33 a4feded9-371a-49d9-82f3-f12c39cce0a7 /tidb/cdc/default/default/upstream/7365852094888221765 {"id":7365852094888221765,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a4feded9-371a-49d9-82f3-f12c39cce0a7 {"id":"a4feded9-371a-49d9-82f3-f12c39cce0a7","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996098} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb75dc33 a4feded9-371a-49d9-82f3-f12c39cce0a7 /tidb/cdc/default/default/upstream/7365852094888221765 {"id":7365852094888221765,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: cd0a6699-d35c-40bc-af3d-73258f65407b Info: {"upstream_id":7365852094888221765,"namespace":"default","id":"cd0a6699-d35c-40bc-af3d-73258f65407b","sink_uri":"blackhole:","create_time":"2024-05-06T19:48:21.882325611+08:00","start_ts":449575938107375619,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575938107375619,"checkpoint_ts":449575938107375619,"checkpoint_time":"2024-05-06 19:48:21.789"} ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc) 3667979264 bytes in 5.84 secs (628207073 bytes/sec) [Pipeline] { [Pipeline] cache VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ef43700014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-245l7, pid:3844, start at 2024-05-06 19:48:25.479976352 +0800 CST m=+5.258526937 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:25.491 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:25.487 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:25.487 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ef43700014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-245l7, pid:3844, start at 2024-05-06 19:48:25.479976352 +0800 CST m=+5.258526937 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:25.491 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:25.487 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:25.487 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ef45b00008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-245l7, pid:3919, start at 2024-05-06 19:48:25.59331589 +0800 CST m=+5.324000632 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:25.601 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:25.580 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:25.580 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_manager/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_manager/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Mon May 6 19:48:30 CST 2024] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.51635165.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:48:33 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bffb70af-f7af-4fbd-a92d-d49b32bcb88a {"id":"bffb70af-f7af-4fbd-a92d-d49b32bcb88a","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996110} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbba89b32 bffb70af-f7af-4fbd-a92d-d49b32bcb88a /tidb/cdc/default/default/upstream/7365852149386214752 {"id":7365852149386214752,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bffb70af-f7af-4fbd-a92d-d49b32bcb88a {"id":"bffb70af-f7af-4fbd-a92d-d49b32bcb88a","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996110} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbba89b32 bffb70af-f7af-4fbd-a92d-d49b32bcb88a /tidb/cdc/default/default/upstream/7365852149386214752 {"id":7365852149386214752,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bffb70af-f7af-4fbd-a92d-d49b32bcb88a {"id":"bffb70af-f7af-4fbd-a92d-d49b32bcb88a","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996110} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbba89b32 bffb70af-f7af-4fbd-a92d-d49b32bcb88a /tidb/cdc/default/default/upstream/7365852149386214752 {"id":7365852149386214752,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.cli.5219.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-7489?protocol=canal-json&enable-tidb-extension=true' -c=ddl-manager Create changefeed successfully! ID: ddl-manager Info: {"upstream_id":7365852149386214752,"namespace":"default","id":"ddl-manager","sink_uri":"file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-7489?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:48:34.048614007+08:00","start_ts":449575941291638789,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575941291638789,"checkpoint_ts":449575941291638789,"checkpoint_time":"2024-05-06 19:48:33.936"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc) 3667979264 bytes in 6.83 secs (537007234 bytes/sec) [Pipeline] { [Pipeline] cache + set +x + workdir=/tmp/tidb_cdc_test/ddl_manager + sink_uri='file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-7489?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/ddl_manager ++ date + echo '[Mon May 6 19:48:35 CST 2024] <<<<<< START storage consumer in ddl_manager case >>>>>>' [Mon May 6 19:48:35 CST 2024] <<<<<< START storage consumer in ddl_manager case >>>>>> + cd /tmp/tidb_cdc_test/ddl_manager + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/ddl_manager + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/ddl_manager/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-7489?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' wait process 5168 exit for 1-th time... wait process 5168 exit for 2-th time... wait process 5168 exit for 3-th time... wait process 5168 exit for 4-th time... wait process 5168 exit for 5-th time... wait process 5168 exit for 6-th time... Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc) 3667979264 bytes in 5.59 secs (655751291 bytes/sec) [Pipeline] { [Pipeline] cache /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (5168) - No such process wait process 5168 exit for 7-th time... process 5168 already exit [Mon May 6 19:48:42 CST 2024] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.53335335.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:48:47 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7365852149386214752 Namespace:default ID:ddl-manager SinkURI:file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-7489?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:48:34.048614007 +0800 CST StartTs:449575941291638789 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002965c20 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575941317853188} {CheckpointTs:449575941711069210 MinTableBarrierTs:449575941711069210 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8d293877-a62a-4990-a15b-6cfa28bbc659 {"id":"8d293877-a62a-4990-a15b-6cfa28bbc659","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996122} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbba89d36 8d293877-a62a-4990-a15b-6cfa28bbc659 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7365852149386214752,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-7489?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:48:34.048614007+08:00","start-ts":449575941291638789,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575941317853188} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":449575941711069210,"min-table-barrier-ts":449575941711069210,"admin-job-type":0} /tidb/cdc/default/default/task/position/8d293877-a62a-4990-a15b-6cfa28bbc659/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852149386214752 {"id":7365852149386214752,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7365852149386214752 Namespace:default ID:ddl-manager SinkURI:file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-7489?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:48:34.048614007 +0800 CST StartTs:449575941291638789 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002965c20 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575941317853188} {CheckpointTs:449575941711069210 MinTableBarrierTs:449575941711069210 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8d293877-a62a-4990-a15b-6cfa28bbc659 {"id":"8d293877-a62a-4990-a15b-6cfa28bbc659","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996122} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbba89d36 8d293877-a62a-4990-a15b-6cfa28bbc659 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7365852149386214752,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-7489?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:48:34.048614007+08:00","start-ts":449575941291638789,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575941317853188} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":449575941711069210,"min-table-barrier-ts":449575941711069210,"admin-job-type":0} /tidb/cdc/default/default/task/position/8d293877-a62a-4990-a15b-6cfa28bbc659/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852149386214752 {"id":7365852149386214752,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7365852149386214752 Namespace:default ID:ddl-manager SinkURI:file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-7489?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:48:34.048614007 +0800 CST StartTs:449575941291638789 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002965c20 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575941317853188} {CheckpointTs:449575941711069210 MinTableBarrierTs:449575941711069210 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8d293877-a62a-4990-a15b-6cfa28bbc659 {"id":"8d293877-a62a-4990-a15b-6cfa28bbc659","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996122} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbba89d36 8d293877-a62a-4990-a15b-6cfa28bbc659 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7365852149386214752,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-7489?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:48:34.048614007+08:00","start-ts":449575941291638789,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575941317853188} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":449575941711069210,"min-table-barrier-ts":449575941711069210,"admin-job-type":0} /tidb/cdc/default/default/task/position/8d293877-a62a-4990-a15b-6cfa28bbc659/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852149386214752 {"id":7365852149386214752,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x [Mon May 6 19:48:47 CST 2024] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.53965398.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > check_lossy_ddl /tmp/tidb_cdc_test/lossy_ddl can't found finish mark run task failed 1-th time, retry later check_lossy_ddl /tmp/tidb_cdc_test/lossy_ddl can't found finish mark run task failed 2-th time, retry later < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:48:51 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7365852149386214752 Namespace:default ID:ddl-manager SinkURI:file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-7489?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:48:34.048614007 +0800 CST StartTs:449575941291638789 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002965c20 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575941317853188} {CheckpointTs:449575941711069210 MinTableBarrierTs:449575941711069210 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8d293877-a62a-4990-a15b-6cfa28bbc659 {"id":"8d293877-a62a-4990-a15b-6cfa28bbc659","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996122} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbba89d36 8d293877-a62a-4990-a15b-6cfa28bbc659 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7365852149386214752,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-7489?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:48:34.048614007+08:00","start-ts":449575941291638789,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575941317853188} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":449575941737283613,"min-table-barrier-ts":449575941737283613,"admin-job-type":0} /tidb/cdc/default/default/task/position/8d293877-a62a-4990-a15b-6cfa28bbc659/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852149386214752 {"id":7365852149386214752,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7365852149386214752 Namespace:default ID:ddl-manager SinkURI:file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-7489?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:48:34.048614007 +0800 CST StartTs:449575941291638789 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002965c20 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575941317853188} {CheckpointTs:449575941711069210 MinTableBarrierTs:449575941711069210 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8d293877-a62a-4990-a15b-6cfa28bbc659 {"id":"8d293877-a62a-4990-a15b-6cfa28bbc659","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996122} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbba89d36 8d293877-a62a-4990-a15b-6cfa28bbc659 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7365852149386214752,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-7489?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:48:34.048614007+08:00","start-ts":449575941291638789,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575941317853188} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":449575941737283613,"min-table-barrier-ts":449575941737283613,"admin-job-type":0} /tidb/cdc/default/default/task/position/8d293877-a62a-4990-a15b-6cfa28bbc659/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852149386214752 {"id":7365852149386214752,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7365852149386214752 Namespace:default ID:ddl-manager SinkURI:file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-7489?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:48:34.048614007 +0800 CST StartTs:449575941291638789 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002965c20 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575941317853188} {CheckpointTs:449575941711069210 MinTableBarrierTs:449575941711069210 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8d293877-a62a-4990-a15b-6cfa28bbc659 {"id":"8d293877-a62a-4990-a15b-6cfa28bbc659","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996122} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbba89d36 8d293877-a62a-4990-a15b-6cfa28bbc659 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7365852149386214752,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-7489?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:48:34.048614007+08:00","start-ts":449575941291638789,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575941317853188} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":449575941737283613,"min-table-barrier-ts":449575941737283613,"admin-job-type":0} /tidb/cdc/default/default/task/position/8d293877-a62a-4990-a15b-6cfa28bbc659/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852149386214752 + grep -q 'etcd info' {"id":7365852149386214752,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x table ddl_manager.finish_mark not exists for 1-th check, retry later check_lossy_ddl /tmp/tidb_cdc_test/lossy_ddl can't found finish mark run task failed 3-th time, retry later table ddl_manager.finish_mark not exists for 2-th check, retry later table ddl_manager.finish_mark not exists for 3-th check, retry later table ddl_manager.finish_mark not exists for 4-th check, retry later check_lossy_ddl /tmp/tidb_cdc_test/lossy_ddl can't found finish mark run task failed 4-th time, retry later table ddl_manager.finish_mark not exists for 5-th check, retry later table ddl_manager.finish_mark not exists for 6-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc) 3667979264 bytes in 17.88 secs (205175210 bytes/sec) [Pipeline] { [Pipeline] cache table ddl_manager.finish_mark not exists for 7-th check, retry later table ddl_manager.finish_mark not exists for 8-th check, retry later check_lossy_ddl /tmp/tidb_cdc_test/lossy_ddl can't found finish mark run task failed 5-th time, retry later table ddl_manager.finish_mark not exists for 9-th check, retry later table ddl_manager.finish_mark not exists for 10-th check, retry later table ddl_manager.finish_mark not exists for 11-th check, retry later table ddl_manager.finish_mark not exists for 12-th check, retry later table ddl_manager.finish_mark not exists for 13-th check, retry later check_lossy_ddl /tmp/tidb_cdc_test/lossy_ddl [2024/05/06 19:48:25.304 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":449575938408579074,\"CommitTs\":449575938408579077,\"RowID\":1,\"PhysicalTableID\":106,\"TableInfo\":{\"id\":106,\"name\":{\"O\":\"example1\",\"L\":\"example1\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":0,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":449575938277507077,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"SchemaID\":104,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example1\",\"TableID\":106,\"IsPartition\":false},\"Version\":449575938277507084,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":65},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":29,\"SplitTxn\":false,\"ReplicatingTs\":449575939011510278,\"HandleKey\":1}"] [2024/05/06 19:48:27.303 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":449575938670723078,\"CommitTs\":449575938670723080,\"RowID\":1,\"PhysicalTableID\":109,\"TableInfo\":{\"id\":109,\"name\":{\"O\":\"example2\",\"L\":\"example2\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":0,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":449575938539651092,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"SchemaID\":104,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example2\",\"TableID\":109,\"IsPartition\":false},\"Version\":449575938539651098,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":65},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":29,\"SplitTxn\":false,\"ReplicatingTs\":449575939535798277,\"HandleKey\":1}"] [2024/05/06 19:48:29.304 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":449575938932867081,\"CommitTs\":449575938932867082,\"RowID\":1,\"PhysicalTableID\":112,\"TableInfo\":{\"id\":112,\"name\":{\"O\":\"example3\",\"L\":\"example3\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":7,\"Flag\":128,\"Flen\":19,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":449575938801795094,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"SchemaID\":104,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example3\",\"TableID\":112,\"IsPartition\":false},\"Version\":449575938801795099,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":65},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":\"2023-04-19 11:48:00\"}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":36,\"SplitTxn\":false,\"ReplicatingTs\":449575940060086278,\"HandleKey\":1}"] [2024/05/06 19:48:31.303 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":449575939208118278,\"CommitTs\":449575939208118279,\"RowID\":1,\"PhysicalTableID\":115,\"TableInfo\":{\"id\":115,\"name\":{\"O\":\"example4\",\"L\":\"example4\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":15,\"Flag\":0,\"Flen\":256,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":449575939077046291,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"SchemaID\":104,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example4\",\"TableID\":115,\"IsPartition\":false},\"Version\":449575939077046295,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":64},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":\"MjAyMy0wNC0xOSAxMTo0ODowMA==\"}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":47,\"SplitTxn\":false,\"ReplicatingTs\":449575940584374277,\"HandleKey\":1}"] [2024/05/06 19:48:33.303 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":449575939470262281,\"CommitTs\":449575939470262282,\"RowID\":1,\"PhysicalTableID\":118,\"TableInfo\":{\"id\":118,\"name\":{\"O\":\"example5\",\"L\":\"example5\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":0,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":449575939339190294,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"SchemaID\":104,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example5\",\"TableID\":118,\"IsPartition\":false},\"Version\":449575939339190299,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":65},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":-1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":29,\"SplitTxn\":false,\"ReplicatingTs\":449575941108924421,\"HandleKey\":1}"] [2024/05/06 19:48:35.303 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":449575939745513476,\"CommitTs\":449575939745513478,\"RowID\":1,\"PhysicalTableID\":121,\"TableInfo\":{\"id\":121,\"name\":{\"O\":\"example6\",\"L\":\"example6\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":0,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":449575939614441484,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"SchemaID\":104,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example6\",\"TableID\":121,\"IsPartition\":false},\"Version\":449575939614441488,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":65},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":-1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":29,\"SplitTxn\":false,\"ReplicatingTs\":449575941632950279,\"HandleKey\":1}"] [2024/05/06 19:48:37.303 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":449575940007919622,\"CommitTs\":449575940007919624,\"RowID\":1,\"PhysicalTableID\":124,\"TableInfo\":{\"id\":124,\"name\":{\"O\":\"example7\",\"L\":\"example7\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":15,\"Flag\":0,\"Flen\":256,\"Decimal\":0,\"Charset\":\"utf8\",\"Collate\":\"utf8_general_ci\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":449575939876585492,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"SchemaID\":104,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example7\",\"TableID\":124,\"IsPartition\":false},\"Version\":449575939876585497,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":64},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":\"MjAyMy0wNC0xOSAxMTo0ODowMA==\"}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":47,\"SplitTxn\":false,\"ReplicatingTs\":449575942157500436,\"HandleKey\":1}"] [2024/05/06 19:48:40.303 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":449575940270063626,\"CommitTs\":449575940270063627,\"RowID\":-1,\"PhysicalTableID\":128,\"TableInfo\":{\"id\":127,\"name\":{\"O\":\"example8\",\"L\":\"example8\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":0,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":449575940138991641,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":{\"type\":1,\"expr\":\"`id`\",\"columns\":[],\"enable\":true,\"is_empty_columns\":false,\"definitions\":[{\"id\":128,\"name\":{\"O\":\"b0\",\"L\":\"b0\"},\"less_than\":[\"0\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":129,\"name\":{\"O\":\"b1\",\"L\":\"b1\"},\"less_than\":[\"MAXVALUE\"],\"in_values\":null,\"policy_ref_info\":null}],\"adding_definitions\":[],\"dropping_definitions\":[],\"NewPartitionIDs\":null,\"states\":null,\"num\":2,\"ddl_state\":0,\"new_table_id\":0,\"ddl_type\":0,\"ddl_expr\":\"\",\"ddl_columns\":null},\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"SchemaID\":104,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example8\",\"TableID\":127,\"IsPartition\":true},\"Version\":449575940151836676,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":65},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":-1},{\"column_id\":2,\"value\":-1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":29,\"SplitTxn\":false,\"ReplicatingTs\":449575942943670277,\"HandleKey\":-1}"] [2024/05/06 19:48:40.303 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":449575940270063627,\"CommitTs\":449575940270063628,\"RowID\":1,\"PhysicalTableID\":129,\"TableInfo\":{\"id\":127,\"name\":{\"O\":\"example8\",\"L\":\"example8\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":0,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":449575940138991641,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":{\"type\":1,\"expr\":\"`id`\",\"columns\":[],\"enable\":true,\"is_empty_columns\":false,\"definitions\":[{\"id\":128,\"name\":{\"O\":\"b0\",\"L\":\"b0\"},\"less_than\":[\"0\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":129,\"name\":{\"O\":\"b1\",\"L\":\"b1\"},\"less_than\":[\"MAXVALUE\"],\"in_values\":null,\"policy_ref_info\":null}],\"adding_definitions\":[],\"dropping_definitions\":[],\"NewPartitionIDs\":null,\"states\":null,\"num\":2,\"ddl_state\":0,\"new_table_id\":0,\"ddl_type\":0,\"ddl_expr\":\"\",\"ddl_columns\":null},\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"SchemaID\":104,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example8\",\"TableID\":127,\"IsPartition\":true},\"Version\":449575940151836676,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":65},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":29,\"SplitTxn\":false,\"ReplicatingTs\":449575942943670281,\"HandleKey\":1}"] [2024/05/06 19:48:42.303 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":449575940545314822,\"CommitTs\":449575940545314823,\"RowID\":1,\"PhysicalTableID\":132,\"TableInfo\":{\"id\":132,\"name\":{\"O\":\"example9\",\"L\":\"example9\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":15,\"Flag\":0,\"Flen\":256,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":449575940413980692,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"SchemaID\":104,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example9\",\"TableID\":132,\"IsPartition\":false},\"Version\":449575940413980696,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":64},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":\"MjAyMy0wNC0xOSAxMTo0ODowMA==\"}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":47,\"SplitTxn\":false,\"ReplicatingTs\":449575943468220421,\"HandleKey\":1}"] [2024/05/06 19:48:44.303 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":449575940807458826,\"CommitTs\":449575940807458827,\"RowID\":1,\"PhysicalTableID\":135,\"TableInfo\":{\"id\":135,\"name\":{\"O\":\"example10\",\"L\":\"example10\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":5,\"Flag\":0,\"Flen\":22,\"Decimal\":-1,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":449575940676124694,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"SchemaID\":104,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example10\",\"TableID\":135,\"IsPartition\":false},\"Version\":449575940676124699,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":65},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":36,\"SplitTxn\":false,\"ReplicatingTs\":449575943992246280,\"HandleKey\":1}"] [2024/05/06 19:48:46.303 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":449575941082447878,\"CommitTs\":449575941082447879,\"RowID\":1,\"PhysicalTableID\":138,\"TableInfo\":{\"id\":138,\"name\":{\"O\":\"example11\",\"L\":\"example11\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":8,\"Flag\":4099,\"Flen\":20,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":8,\"Flag\":0,\"Flen\":20,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":449575940951375888,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"SchemaID\":104,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example11\",\"TableID\":138,\"IsPartition\":false},\"Version\":449575940951375893,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":65},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":29,\"SplitTxn\":false,\"ReplicatingTs\":449575944516796421,\"HandleKey\":1}"] [2024/05/06 19:48:48.303 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":449575941344591881,\"CommitTs\":449575941344591882,\"RowID\":1,\"PhysicalTableID\":141,\"TableInfo\":{\"id\":141,\"name\":{\"O\":\"example12\",\"L\":\"example12\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":251,\"Flag\":0,\"Flen\":4294967295,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":449575941213782037,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"SchemaID\":104,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example12\",\"TableID\":141,\"IsPartition\":false},\"Version\":449575941213782041,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":64},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":\"MjAyMy0wNC0xOSAxMTo0ODowMA==\"}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":47,\"SplitTxn\":false,\"ReplicatingTs\":449575945041084421,\"HandleKey\":1}"] [2024/05/06 19:48:50.303 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":449575941619843076,\"CommitTs\":449575941619843077,\"RowID\":1,\"PhysicalTableID\":144,\"TableInfo\":{\"id\":144,\"name\":{\"O\":\"example13\",\"L\":\"example13\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":247,\"Flag\":0,\"Flen\":1,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":[\"a\",\"b\",\"c\"],\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":449575941488771079,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"SchemaID\":104,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example13\",\"TableID\":144,\"IsPartition\":false},\"Version\":449575941488771083,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":64},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":29,\"SplitTxn\":false,\"ReplicatingTs\":449575945565110279,\"HandleKey\":1}"] [2024/05/06 19:48:52.303 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":449575941881987078,\"CommitTs\":449575941881987079,\"RowID\":1,\"PhysicalTableID\":147,\"TableInfo\":{\"id\":147,\"name\":{\"O\":\"example14\",\"L\":\"example14\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":15,\"Flag\":0,\"Flen\":256,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":449575941750915093,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"SchemaID\":104,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example14\",\"TableID\":147,\"IsPartition\":false},\"Version\":449575941750915097,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":64},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":\"MjAyMy0wNC0xOSAxMTo0ODowMA==\"}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":47,\"SplitTxn\":false,\"ReplicatingTs\":449575946089660421,\"HandleKey\":1}"] [2024/05/06 19:48:54.403 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":449575942144131080,\"CommitTs\":449575942144131081,\"RowID\":1,\"PhysicalTableID\":150,\"TableInfo\":{\"id\":150,\"name\":{\"O\":\"example15\",\"L\":\"example15\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2},{\"id\":2,\"name\":{\"O\":\"b\",\"L\":\"b\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":0,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":2,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":449575942013059095,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"SchemaID\":104,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example15\",\"TableID\":150,\"IsPartition\":false},\"Version\":449575942026166273,\"RowColumnsOffset\":{\"1\":0,\"2\":1},\"ColumnsFlag\":{\"1\":11,\"2\":65},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1},{\"column_id\":2,\"value\":-1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":29,\"SplitTxn\":false,\"ReplicatingTs\":449575946626793475,\"HandleKey\":1}"] [2024/05/06 19:48:56.403 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":449575942419382279,\"CommitTs\":449575942419382280,\"RowID\":1,\"PhysicalTableID\":154,\"TableInfo\":{\"id\":153,\"name\":{\"O\":\"example16\",\"L\":\"example16\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"a\",\"L\":\"a\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":1,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":449575942288572437,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":{\"type\":1,\"expr\":\"`a`\",\"columns\":[],\"enable\":true,\"is_empty_columns\":false,\"definitions\":[{\"id\":154,\"name\":{\"O\":\"p0\",\"L\":\"p0\"},\"less_than\":[\"6\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":155,\"name\":{\"O\":\"p1\",\"L\":\"p1\"},\"less_than\":[\"11\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":156,\"name\":{\"O\":\"p2\",\"L\":\"p2\"},\"less_than\":[\"21\"],\"in_values\":null,\"policy_ref_info\":null}],\"adding_definitions\":[],\"dropping_definitions\":[],\"NewPartitionIDs\":null,\"states\":null,\"num\":3,\"ddl_state\":0,\"new_table_id\":0,\"ddl_type\":0,\"ddl_expr\":\"\",\"ddl_columns\":null},\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"SchemaID\":104,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example16\",\"TableID\":153,\"IsPartition\":true},\"Version\":449575942288572442,\"RowColumnsOffset\":{\"1\":0},\"ColumnsFlag\":{\"1\":11},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":25,\"SplitTxn\":false,\"ReplicatingTs\":449575947151081475,\"HandleKey\":1}"] [2024/05/06 19:48:58.303 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":449575942550716423,\"CommitTs\":449575942550716424,\"RowID\":18,\"PhysicalTableID\":158,\"TableInfo\":{\"id\":158,\"name\":{\"O\":\"example17\",\"L\":\"example17\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"a\",\"L\":\"a\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":1,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":449575942419382307,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"SchemaID\":104,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example17\",\"TableID\":158,\"IsPartition\":false},\"Version\":449575942419382313,\"RowColumnsOffset\":{\"1\":0},\"ColumnsFlag\":{\"1\":11},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":18}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":25,\"SplitTxn\":false,\"ReplicatingTs\":449575947662524424,\"HandleKey\":18}"] [2024/05/06 19:49:00.303 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":449575942825967620,\"CommitTs\":449575942825967621,\"RowID\":1,\"PhysicalTableID\":162,\"TableInfo\":{\"id\":161,\"name\":{\"O\":\"example18\",\"L\":\"example18\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":1,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":449575942694633481,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":{\"type\":1,\"expr\":\"`id`\",\"columns\":[],\"enable\":true,\"is_empty_columns\":false,\"definitions\":[{\"id\":162,\"name\":{\"O\":\"pBefore1950\",\"L\":\"pbefore1950\"},\"less_than\":[\"1950\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":163,\"name\":{\"O\":\"p1950\",\"L\":\"p1950\"},\"less_than\":[\"1960\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":164,\"name\":{\"O\":\"p1960\",\"L\":\"p1960\"},\"less_than\":[\"1970\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":165,\"name\":{\"O\":\"p1970\",\"L\":\"p1970\"},\"less_than\":[\"1980\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":166,\"name\":{\"O\":\"p1980\",\"L\":\"p1980\"},\"less_than\":[\"1990\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":167,\"name\":{\"O\":\"p1990\",\"L\":\"p1990\"},\"less_than\":[\"2000\"],\"in_values\":null,\"policy_ref_info\":null}],\"adding_definitions\":[],\"dropping_definitions\":[],\"NewPartitionIDs\":null,\"states\":null,\"num\":6,\"ddl_state\":0,\"new_table_id\":0,\"ddl_type\":0,\"ddl_expr\":\"\",\"ddl_columns\":null},\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"SchemaID\":104,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example18\",\"TableID\":161,\"IsPartition\":true},\"Version\":449575942694633486,\"RowColumnsOffset\":{\"1\":0},\"ColumnsFlag\":{\"1\":11},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":25,\"SplitTxn\":false,\"ReplicatingTs\":449575948186550282,\"HandleKey\":1}"] [2024/05/06 19:49:00.304 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":449575942825967620,\"CommitTs\":449575942825967621,\"RowID\":1977,\"PhysicalTableID\":165,\"TableInfo\":{\"id\":161,\"name\":{\"O\":\"example18\",\"L\":\"example18\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":1,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":449575942694633481,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":{\"type\":1,\"expr\":\"`id`\",\"columns\":[],\"enable\":true,\"is_empty_columns\":false,\"definitions\":[{\"id\":162,\"name\":{\"O\":\"pBefore1950\",\"L\":\"pbefore1950\"},\"less_than\":[\"1950\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":163,\"name\":{\"O\":\"p1950\",\"L\":\"p1950\"},\"less_than\":[\"1960\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":164,\"name\":{\"O\":\"p1960\",\"L\":\"p1960\"},\"less_than\":[\"1970\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":165,\"name\":{\"O\":\"p1970\",\"L\":\"p1970\"},\"less_than\":[\"1980\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":166,\"name\":{\"O\":\"p1980\",\"L\":\"p1980\"},\"less_than\":[\"1990\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":167,\"name\":{\"O\":\"p1990\",\"L\":\"p1990\"},\"less_than\":[\"2000\"],\"in_values\":null,\"policy_ref_info\":null}],\"adding_definitions\":[],\"dropping_definitions\":[],\"NewPartitionIDs\":null,\"states\":null,\"num\":6,\"ddl_state\":0,\"new_table_id\":0,\"ddl_type\":0,\"ddl_expr\":\"\",\"ddl_columns\":null},\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"SchemaID\":104,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example18\",\"TableID\":161,\"IsPartition\":true},\"Version\":449575942694633486,\"RowColumnsOffset\":{\"1\":0},\"ColumnsFlag\":{\"1\":11},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1977}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":25,\"SplitTxn\":false,\"ReplicatingTs\":449575948186550285,\"HandleKey\":1977}"] [2024/05/06 19:49:01.403 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":449575942825967620,\"CommitTs\":449575942825967621,\"RowID\":1999,\"PhysicalTableID\":167,\"TableInfo\":{\"id\":161,\"name\":{\"O\":\"example18\",\"L\":\"example18\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":1,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":449575942694633481,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":{\"type\":1,\"expr\":\"`id`\",\"columns\":[],\"enable\":true,\"is_empty_columns\":false,\"definitions\":[{\"id\":162,\"name\":{\"O\":\"pBefore1950\",\"L\":\"pbefore1950\"},\"less_than\":[\"1950\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":163,\"name\":{\"O\":\"p1950\",\"L\":\"p1950\"},\"less_than\":[\"1960\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":164,\"name\":{\"O\":\"p1960\",\"L\":\"p1960\"},\"less_than\":[\"1970\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":165,\"name\":{\"O\":\"p1970\",\"L\":\"p1970\"},\"less_than\":[\"1980\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":166,\"name\":{\"O\":\"p1980\",\"L\":\"p1980\"},\"less_than\":[\"1990\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":167,\"name\":{\"O\":\"p1990\",\"L\":\"p1990\"},\"less_than\":[\"2000\"],\"in_values\":null,\"policy_ref_info\":null}],\"adding_definitions\":[],\"dropping_definitions\":[],\"NewPartitionIDs\":null,\"states\":null,\"num\":6,\"ddl_state\":0,\"new_table_id\":0,\"ddl_type\":0,\"ddl_expr\":\"\",\"ddl_columns\":null},\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"SchemaID\":104,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example18\",\"TableID\":161,\"IsPartition\":true},\"Version\":449575942694633486,\"RowColumnsOffset\":{\"1\":0},\"ColumnsFlag\":{\"1\":11},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1999}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":25,\"SplitTxn\":false,\"ReplicatingTs\":449575948461801475,\"HandleKey\":1999}"] [2024/05/06 19:49:07.403 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":449575943349993481,\"CommitTs\":449575943349993482,\"RowID\":2003,\"PhysicalTableID\":174,\"TableInfo\":{\"id\":161,\"name\":{\"O\":\"example18\",\"L\":\"example18\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":null,\"constraint_info\":null,\"fk_info\":null,\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":1,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":449575943245135878,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":{\"type\":1,\"expr\":\"`id`\",\"columns\":null,\"enable\":true,\"is_empty_columns\":false,\"definitions\":[{\"id\":169,\"name\":{\"O\":\"pBefore1960\",\"L\":\"pbefore1960\"},\"less_than\":[\"1960\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":164,\"name\":{\"O\":\"p1960\",\"L\":\"p1960\"},\"less_than\":[\"1970\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":165,\"name\":{\"O\":\"p1970\",\"L\":\"p1970\"},\"less_than\":[\"1980\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":172,\"name\":{\"O\":\"p1980\",\"L\":\"p1980\"},\"less_than\":[\"1990\"],\"in_values\":null,\"policy_ref_info\":null},{\"id\":174,\"name\":{\"O\":\"p1990to2010\",\"L\":\"p1990to2010\"},\"less_than\":[\"2010\"],\"in_values\":null,\"policy_ref_info\":null}],\"adding_definitions\":null,\"dropping_definitions\":null,\"NewPartitionIDs\":null,\"states\":null,\"num\":5,\"ddl_state\":0,\"new_table_id\":0,\"ddl_type\":1,\"ddl_expr\":\"`id`\",\"ddl_columns\":null},\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"SchemaID\":104,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example18\",\"TableID\":161,\"IsPartition\":true},\"Version\":449575943245135883,\"RowColumnsOffset\":{\"1\":0},\"ColumnsFlag\":{\"1\":11},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":2003}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":25,\"SplitTxn\":false,\"ReplicatingTs\":449575950034665475,\"HandleKey\":2003}"] [2024/05/06 19:49:11.403 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"StartTs\":449575944280604678,\"CommitTs\":449575944280604679,\"RowID\":1,\"PhysicalTableID\":182,\"TableInfo\":{\"id\":182,\"name\":{\"O\":\"example19\",\"L\":\"example19\"},\"charset\":\"utf8mb4\",\"collate\":\"utf8mb4_bin\",\"cols\":[{\"id\":1,\"name\":{\"O\":\"a\",\"L\":\"a\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2}],\"index_info\":[],\"constraint_info\":null,\"fk_info\":[],\"state\":5,\"pk_is_handle\":true,\"is_common_handle\":false,\"common_handle_version\":0,\"comment\":\"\",\"auto_inc_id\":0,\"auto_id_cache\":0,\"auto_rand_id\":0,\"max_col_id\":1,\"max_idx_id\":0,\"max_fk_id\":0,\"max_cst_id\":0,\"update_timestamp\":449575944149794837,\"ShardRowIDBits\":0,\"max_shard_row_id_bits\":0,\"auto_random_bits\":0,\"auto_random_range_bits\":0,\"pre_split_regions\":0,\"partition\":null,\"compression\":\"\",\"view\":null,\"sequence\":null,\"Lock\":null,\"version\":5,\"tiflash_replica\":null,\"is_columnar\":false,\"temp_table_type\":0,\"cache_table_status\":0,\"policy_ref_info\":null,\"stats_options\":null,\"exchange_partition_info\":null,\"ttl_info\":null,\"SchemaID\":104,\"TableName\":{\"Schema\":\"lossy_ddl\",\"Table\":\"example19\",\"TableID\":182,\"IsPartition\":false},\"Version\":449575944149794841,\"RowColumnsOffset\":{\"1\":0},\"ColumnsFlag\":{\"1\":11},\"HandleIndexID\":-1,\"IndexColumnsOffset\":[[0]]},\"Columns\":[{\"column_id\":1,\"value\":1}],\"PreColumns\":null,\"Checksum\":null,\"ApproximateDataSize\":25,\"SplitTxn\":false,\"ReplicatingTs\":449575951083503619,\"HandleKey\":1}"] run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... table ddl_manager.finish_mark not exists for 14-th check, retry later wait process cdc.test exit for 3-th time... process cdc.test already exit [Mon May 6 19:49:17 CST 2024] <<<<<< run test case lossy_ddl success! >>>>>> table ddl_manager.finish_mark not exists for 15-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc) 3667979264 bytes in 15.73 secs (233150653 bytes/sec) [Pipeline] { [Pipeline] sh [Pipeline] sh [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G02 + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G03 Run cases: row_format drop_many_tables processor_stop_delay partition_table PROW_JOB_ID=8fa694ce-ccab-4faa-bb8b-f95ef2091797 JENKINS_NODE_COOKIE=a122db0a-f0cb-47c0-a36e-64449d08776e BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/ GOLANG_VERSION=1.21.6 HOSTNAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-t7zm9 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test","buildid":"1787445299229233156","prowjobid":"8fa694ce-ccab-4faa-bb8b-f95ef2091797","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#61 TEST_GROUP=G03 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233156 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-0crks GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-81bc5d443e840859e7b2b4bd4bb8be00446de8a572b18ada90bce15670edaaa5 NODE_LABELS=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-0crks p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-t7zm9 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-t7zm9 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=61 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/row_format/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh Run cases: consistent_replicate_ddl consistent_replicate_gbk consistent_replicate_nfs consistent_replicate_storage_file consistent_replicate_storage_file_large_value consistent_replicate_storage_s3 consistent_partition_table kafka_big_messages_v2 multi_tables_ddl_v2 multi_topics_v2 storage_cleanup csv_storage_basic csv_storage_multi_tables_ddl csv_storage_partition_table PROW_JOB_ID=8fa694ce-ccab-4faa-bb8b-f95ef2091797 JENKINS_NODE_COOKIE=f549d278-3abb-49a4-855b-828dcb74693c BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/ GOLANG_VERSION=1.21.6 HOSTNAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-vs88z HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test","buildid":"1787445299229233156","prowjobid":"8fa694ce-ccab-4faa-bb8b-f95ef2091797","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#61 TEST_GROUP=G02 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233156 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-bd64b GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-81bc5d443e840859e7b2b4bd4bb8be00446de8a572b18ada90bce15670edaaa5 NODE_LABELS=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-bd64b p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-vs88z GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-vs88z GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=61 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/consistent_replicate_ddl/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:49:21 CST 2024] <<<<<< run test case consistent_replicate_ddl success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G01 Run cases: http_api http_api_tls api_v2 http_api_tls_with_user_auth cli_tls_with_auth kafka_simple_basic kafka_simple_basic_avro kafka_simple_handle_key_only kafka_simple_handle_key_only_avro kafka_simple_claim_check kafka_simple_claim_check_avro canal_json_adapter_compatibility canal_json_basic canal_json_content_compatible multi_topics avro_basic canal_json_handle_key_only open_protocol_handle_key_only canal_json_claim_check open_protocol_claim_check canal_json_storage_basic canal_json_storage_partition_table multi_tables_ddl PROW_JOB_ID=8fa694ce-ccab-4faa-bb8b-f95ef2091797 JENKINS_NODE_COOKIE=b9f58c53-46cb-48ca-a076-f34e9f3e348a BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/ GOLANG_VERSION=1.21.6 HOSTNAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-b8w74 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test","buildid":"1787445299229233156","prowjobid":"8fa694ce-ccab-4faa-bb8b-f95ef2091797","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#61 TEST_GROUP=G01 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233156 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-52z2z GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-81bc5d443e840859e7b2b4bd4bb8be00446de8a572b18ada90bce15670edaaa5 NODE_LABELS=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-b8w74 p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-52z2z GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-b8w74 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=61 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/http_api/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:49:21 CST 2024] <<<<<< run test case http_api success! >>>>>> table ddl_manager.finish_mark not exists for 16-th check, retry later [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G08 Run cases: processor_err_chan changefeed_reconstruct multi_capture synced_status_with_redo PROW_JOB_ID=8fa694ce-ccab-4faa-bb8b-f95ef2091797 JENKINS_NODE_COOKIE=a8a89061-6640-4e9f-a1c9-00636325131c BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/ GOLANG_VERSION=1.21.6 HOSTNAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test","buildid":"1787445299229233156","prowjobid":"8fa694ce-ccab-4faa-bb8b-f95ef2091797","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#61 TEST_GROUP=G08 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233156 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-rs3q1 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-81bc5d443e840859e7b2b4bd4bb8be00446de8a572b18ada90bce15670edaaa5 NODE_LABELS=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-rs3q1 p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=61 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env [Pipeline] sh find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/processor_err_chan/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G09 Run cases: gc_safepoint changefeed_pause_resume cli_with_auth savepoint synced_status PROW_JOB_ID=8fa694ce-ccab-4faa-bb8b-f95ef2091797 JENKINS_NODE_COOKIE=fbc567c8-a015-463d-9611-9e3ae9c1270d BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/ GOLANG_VERSION=1.21.6 HOSTNAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test","buildid":"1787445299229233156","prowjobid":"8fa694ce-ccab-4faa-bb8b-f95ef2091797","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#61 TEST_GROUP=G09 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233156 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-whxv2 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-81bc5d443e840859e7b2b4bd4bb8be00446de8a572b18ada90bce15670edaaa5 NODE_LABELS=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-whxv2 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=61 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/gc_safepoint/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G04 Run cases: foreign_key ddl_puller_lag ddl_only_block_related_table changefeed_auto_stop PROW_JOB_ID=8fa694ce-ccab-4faa-bb8b-f95ef2091797 JENKINS_NODE_COOKIE=1b5d97bc-8939-495a-92ed-248b5bebff86 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/ GOLANG_VERSION=1.21.6 HOSTNAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-nrrkt HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test","buildid":"1787445299229233156","prowjobid":"8fa694ce-ccab-4faa-bb8b-f95ef2091797","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#61 TEST_GROUP=G04 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233156 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-b3pdh GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-81bc5d443e840859e7b2b4bd4bb8be00446de8a572b18ada90bce15670edaaa5 NODE_LABELS=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-nrrkt p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-b3pdh GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-nrrkt GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=61 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/foreign_key/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G07 Run cases: kv_client_stream_reconnect cdc split_region PROW_JOB_ID=8fa694ce-ccab-4faa-bb8b-f95ef2091797 JENKINS_NODE_COOKIE=656a114b-0280-47e7-acef-938741097274 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/ GOLANG_VERSION=1.21.6 HOSTNAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w9f11 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test","buildid":"1787445299229233156","prowjobid":"8fa694ce-ccab-4faa-bb8b-f95ef2091797","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#61 TEST_GROUP=G07 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233156 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-0w7kh GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-81bc5d443e840859e7b2b4bd4bb8be00446de8a572b18ada90bce15670edaaa5 NODE_LABELS=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w9f11 p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-0w7kh GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w9f11 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=61 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kv_client_stream_reconnect/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G06 Run cases: sink_retry changefeed_error ddl_sequence resourcecontrol PROW_JOB_ID=8fa694ce-ccab-4faa-bb8b-f95ef2091797 JENKINS_NODE_COOKIE=64a3fee8-d25b-46e9-95f6-dd4088148ca3 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/ GOLANG_VERSION=1.21.6 HOSTNAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-drnnn HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test","buildid":"1787445299229233156","prowjobid":"8fa694ce-ccab-4faa-bb8b-f95ef2091797","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#61 TEST_GROUP=G06 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233156 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-z2m3f GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-81bc5d443e840859e7b2b4bd4bb8be00446de8a572b18ada90bce15670edaaa5 NODE_LABELS=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-drnnn p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-z2m3f GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-drnnn GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=61 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/sink_retry/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table ddl_manager.finish_mark not exists for 17-th check, retry later [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/http_api_tls/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:49:25 CST 2024] <<<<<< run test case http_api_tls success! >>>>>> [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { table ddl_manager.finish_mark not exists for 18-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/row_format Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/processor_err_chan Starting Upstream PD... find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/consistent_replicate_gbk/run.sh using Sink-Type: storage... <<================= * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:24927; Connection refused * Closing connection 0 Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { You are running an older version of MinIO released 3 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend start tidb cluster in /tmp/tidb_cdc_test/kv_client_stream_reconnect Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) start tidb cluster in /tmp/tidb_cdc_test/foreign_key Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { table ddl_manager.finish_mark not exists for 19-th check, retry later [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:24927 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 17CCE3EDA888F250 < X-Xss-Protection: 1; mode=block < Date: Mon, 06 May 2024 11:49:27 GMT < { [data not shown] * Connection #0 to host 127.0.0.1 left intact [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow [Pipeline] { Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow Bucket 's3://logbucket/' created [Mon May 6 19:49:28 CST 2024] <<<<<< run test case consistent_replicate_gbk success! >>>>>> Exiting on signal: INTERRUPT [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] cache start tidb cluster in /tmp/tidb_cdc_test/sink_retry Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/api_v2/run.sh using Sink-Type: storage... <<================= Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table ddl_manager.finish_mark not exists for 20-th check, retry later Starting Upstream TiDB... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_csv_update/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/gc_safepoint Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 21-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/http_api_tls_with_user_auth/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:49:32 CST 2024] <<<<<< run test case http_api_tls_with_user_auth success! >>>>>> Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/storage_csv_update Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 22-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/consistent_replicate_nfs/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:49:34 CST 2024] <<<<<< run test case consistent_replicate_nfs success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 23-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/cli_tls_with_auth/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 24-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f397440010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l, pid:1293, start at 2024-05-06 19:49:36.350347312 +0800 CST m=+5.359564564 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:36.363 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:36.337 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:36.337 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f397440010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l, pid:1293, start at 2024-05-06 19:49:36.350347312 +0800 CST m=+5.359564564 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:36.363 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:36.337 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:36.337 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f3aac00013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l, pid:1374, start at 2024-05-06 19:49:37.607536927 +0800 CST m=+6.563140316 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:37.616 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:37.584 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:37.584 +0800 All versions after safe point can be accessed. (DO NOT EDIT) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/consistent_replicate_storage_file/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:49:37 CST 2024] <<<<<< run test case consistent_replicate_storage_file success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_err_chan/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_err_chan/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f38c08000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-t7zm9, pid:1233, start at 2024-05-06 19:49:35.630286178 +0800 CST m=+5.456739935 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:35.637 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:35.618 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:35.618 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f38c08000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-t7zm9, pid:1233, start at 2024-05-06 19:49:35.630286178 +0800 CST m=+5.456739935 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:35.637 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:35.618 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:35.618 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f38d980013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-t7zm9, pid:1302, start at 2024-05-06 19:49:35.738025813 +0800 CST m=+5.503931938 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:35.747 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:35.718 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:35.718 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f3b1e00013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-drnnn, pid:1238, start at 2024-05-06 19:49:38.086346446 +0800 CST m=+5.204594765 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:38.095 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:38.090 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:38.090 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f3b1e00013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-drnnn, pid:1238, start at 2024-05-06 19:49:38.086346446 +0800 CST m=+5.204594765 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:38.095 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:38.090 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:38.090 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f3b1c80014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-drnnn, pid:1312, start at 2024-05-06 19:49:38.063705546 +0800 CST m=+5.134535592 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:38.070 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:38.034 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:38.034 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sink_retry/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sink_retry/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f3b4100006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w9f11, pid:1181, start at 2024-05-06 19:49:38.186719636 +0800 CST m=+5.262623465 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:38.193 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:38.180 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:38.180 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/row_format/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/row_format/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/row_format/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/row_format/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/row_format/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f39ec80012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-nrrkt, pid:1223, start at 2024-05-06 19:49:36.849077843 +0800 CST m=+5.427842957 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:36.859 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:36.818 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:36.818 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f39ec80012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-nrrkt, pid:1223, start at 2024-05-06 19:49:36.849077843 +0800 CST m=+5.427842957 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:36.859 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:36.818 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:36.818 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f3a0680014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-nrrkt, pid:1297, start at 2024-05-06 19:49:36.971559066 +0800 CST m=+5.491407496 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:36.981 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:36.972 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:36.972 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/foreign_key/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/foreign_key/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc) 3667979264 bytes in 9.43 secs (389089578 bytes/sec) [Pipeline] { [Pipeline] cache ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 25-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/consistent_replicate_storage_file_large_value/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:49:41 CST 2024] <<<<<< run test case consistent_replicate_storage_file_large_value success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f3b4100006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w9f11, pid:1181, start at 2024-05-06 19:49:38.186719636 +0800 CST m=+5.262623465 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:38.193 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:38.180 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:38.180 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f3b59c0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w9f11, pid:1266, start at 2024-05-06 19:49:38.323792755 +0800 CST m=+5.337260934 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:38.330 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:38.330 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:38.330 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.cli.2624.out cli tso query --pd=http://127.0.0.1:2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.cli.2604.out cli tso query --pd=http://127.0.0.1:2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.cli.2627.out cli tso query --pd=http://127.0.0.1:2379 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f3e3cc0011 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz, pid:1279, start at 2024-05-06 19:49:41.255220766 +0800 CST m=+5.410854735 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:41.262 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:41.235 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:41.235 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 26-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + tso='449575959013621761 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575959013621761 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + set +x + tso='449575959060283395 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575959060283395 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:49:43 CST 2024] <<<<<< START cdc server in row_format case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.26692671.out server --log-file /tmp/tidb_cdc_test/row_format/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/row_format/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x + tso='449575959165140993 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575959165140993 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:49:43 CST 2024] <<<<<< START cdc server in foreign_key case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.26482650.out server --log-file /tmp/tidb_cdc_test/foreign_key/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/foreign_key/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Mon May 6 19:49:43 CST 2024] <<<<<< START cdc server in kv_client_stream_reconnect case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/kv/kvClientForceReconnect=return(true)' + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kv_client_stream_reconnect.26572659.out server --log-file /tmp/tidb_cdc_test/kv_client_stream_reconnect/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kv_client_stream_reconnect/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 start tidb cluster in /tmp/tidb_cdc_test/cli_tls_with_auth Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... table ddl_manager.finish_mark not exists for 27-th check, retry later ***************** properties ***************** "operationcount"="0" "scanproportion"="0" "readproportion"="0" "mysql.port"="4000" "readallfields"="true" "mysql.host"="127.0.0.1" "recordcount"="10" "mysql.user"="root" "updateproportion"="0" "threadcount"="2" "workload"="core" "dotransactions"="false" "insertproportion"="0" "requestdistribution"="uniform" "mysql.db"="sink_retry" ********************************************** Run finished, takes 11.000521ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1710.6, Avg(us): 2112, Min(us): 1020, Max(us): 5076, 95th(us): 6000, 99th(us): 6000 [Mon May 6 19:49:44 CST 2024] <<<<<< START cdc server in sink_retry case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkTxnRandomError=25%return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.26862688.out server --log-file /tmp/tidb_cdc_test/sink_retry/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sink_retry/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f3e3cc0011 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz, pid:1279, start at 2024-05-06 19:49:41.255220766 +0800 CST m=+5.410854735 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:41.262 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:41.235 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:41.235 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f3e5440006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz, pid:1362, start at 2024-05-06 19:49:41.335553869 +0800 CST m=+5.415970551 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:41.342 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:41.329 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:41.329 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/gc_safepoint/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/gc_safepoint/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f40e900007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-px7vh, pid:4897, start at 2024-05-06 19:49:43.978407506 +0800 CST m=+5.158890855 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:43.985 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:43.972 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:43.972 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f40e900007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-px7vh, pid:4897, start at 2024-05-06 19:49:43.978407506 +0800 CST m=+5.158890855 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:43.985 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:43.972 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:43.972 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f4101c0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-px7vh, pid:4979, start at 2024-05-06 19:49:44.119091266 +0800 CST m=+5.242168225 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:44.125 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:44.121 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:44.121 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/storage_csv_update/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/storage_csv_update/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/storage_csv_update/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/storage_csv_update/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/storage_csv_update/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table ddl_manager.finish_mark not exists for 28-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > [Mon May 6 19:49:46 CST 2024] <<<<<< START cdc server in gc_safepoint case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/txnutil/gc/InjectGcSafepointUpdateInterval=return(500)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.gc_safepoint.26042606.out server --log-file /tmp/tidb_cdc_test/gc_safepoint/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/gc_safepoint/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:49:46 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cedafa47-e99f-43fa-86fd-eb5bda5bb70d {"id":"cedafa47-e99f-43fa-86fd-eb5bda5bb70d","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996183} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbcb9c63e cedafa47-e99f-43fa-86fd-eb5bda5bb70d /tidb/cdc/default/default/upstream/7365852452040399145 {"id":7365852452040399145,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cedafa47-e99f-43fa-86fd-eb5bda5bb70d {"id":"cedafa47-e99f-43fa-86fd-eb5bda5bb70d","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996183} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbcb9c63e cedafa47-e99f-43fa-86fd-eb5bda5bb70d /tidb/cdc/default/default/upstream/7365852452040399145 {"id":7365852452040399145,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cedafa47-e99f-43fa-86fd-eb5bda5bb70d {"id":"cedafa47-e99f-43fa-86fd-eb5bda5bb70d","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996183} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbcb9c63e cedafa47-e99f-43fa-86fd-eb5bda5bb70d /tidb/cdc/default/default/upstream/7365852452040399145 {"id":7365852452040399145,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.cli.2724.out cli changefeed create --start-ts=449575959060283395 '--sink-uri=file:///tmp/tidb_cdc_test/row_format/storage_test/ticdc-row-format-test-9545?protocol=canal-json&enable-tidb-extension=true' < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:49:46 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4fa91fe0-2c91-46a9-914d-715153067f2d {"id":"4fa91fe0-2c91-46a9-914d-715153067f2d","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996183} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbcc0a82f 4fa91fe0-2c91-46a9-914d-715153067f2d /tidb/cdc/default/default/upstream/7365852461325051241 {"id":7365852461325051241,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4fa91fe0-2c91-46a9-914d-715153067f2d {"id":"4fa91fe0-2c91-46a9-914d-715153067f2d","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996183} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbcc0a82f 4fa91fe0-2c91-46a9-914d-715153067f2d /tidb/cdc/default/default/upstream/7365852461325051241 {"id":7365852461325051241,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4fa91fe0-2c91-46a9-914d-715153067f2d {"id":"4fa91fe0-2c91-46a9-914d-715153067f2d","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996183} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbcc0a82f 4fa91fe0-2c91-46a9-914d-715153067f2d /tidb/cdc/default/default/upstream/7365852461325051241 {"id":7365852461325051241,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + workdir=/tmp/tidb_cdc_test/kv_client_stream_reconnect + sink_uri='file:///tmp/tidb_cdc_test/kv_client_stream_reconnect/storage_test/kv-client-stream-reconnect-934?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/kv_client_stream_reconnect ++ date + echo '[Mon May 6 19:49:46 CST 2024] <<<<<< START storage consumer in kv_client_stream_reconnect case >>>>>>' [Mon May 6 19:49:46 CST 2024] <<<<<< START storage consumer in kv_client_stream_reconnect case >>>>>> + cd /tmp/tidb_cdc_test/kv_client_stream_reconnect + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/kv_client_stream_reconnect + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/kv_client_stream_reconnect/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/kv_client_stream_reconnect/storage_test/kv-client-stream-reconnect-934?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' Create changefeed successfully! ID: a561b66e-b288-44ea-9bc9-14e0a46a94e9 Info: {"upstream_id":7365852452040399145,"namespace":"default","id":"a561b66e-b288-44ea-9bc9-14e0a46a94e9","sink_uri":"file:///tmp/tidb_cdc_test/row_format/storage_test/ticdc-row-format-test-9545?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:49:46.818005593+08:00","start_ts":449575959060283395,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575959060283395,"checkpoint_ts":449575959060283395,"checkpoint_time":"2024-05-06 19:49:41.718"} PASS + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:49:46 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c5ff30bd-2a02-4171-81e8-34aa22323c92 {"id":"c5ff30bd-2a02-4171-81e8-34aa22323c92","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996184} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbcc09c3c c5ff30bd-2a02-4171-81e8-34aa22323c92 /tidb/cdc/default/default/upstream/7365852454402010531 {"id":7365852454402010531,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c5ff30bd-2a02-4171-81e8-34aa22323c92 {"id":"c5ff30bd-2a02-4171-81e8-34aa22323c92","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996184} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbcc09c3c c5ff30bd-2a02-4171-81e8-34aa22323c92 /tidb/cdc/default/default/upstream/7365852454402010531 {"id":7365852454402010531,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c5ff30bd-2a02-4171-81e8-34aa22323c92 {"id":"c5ff30bd-2a02-4171-81e8-34aa22323c92","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996184} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbcc09c3c c5ff30bd-2a02-4171-81e8-34aa22323c92 /tidb/cdc/default/default/upstream/7365852454402010531 {"id":7365852454402010531,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.cli.2707.out cli changefeed create --start-ts=449575959165140993 '--sink-uri=file:///tmp/tidb_cdc_test/foreign_key/storage_test/ticdc-foreign-key-test-20237?protocol=canal-json&enable-tidb-extension=true' coverage: 2.7% of statements in github.com/pingcap/tiflow/... Create changefeed successfully! ID: 87aa183f-8223-49b8-ad93-03557eba9cb5 Info: {"upstream_id":7365852454402010531,"namespace":"default","id":"87aa183f-8223-49b8-ad93-03557eba9cb5","sink_uri":"file:///tmp/tidb_cdc_test/foreign_key/storage_test/ticdc-foreign-key-test-20237?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:49:47.176219746+08:00","start_ts":449575959165140993,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575959165140993,"checkpoint_ts":449575959165140993,"checkpoint_time":"2024-05-06 19:49:42.118"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:49:47 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2fdba142-40f8-4ae5-985b-c54d2f20231f {"id":"2fdba142-40f8-4ae5-985b-c54d2f20231f","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996184} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbcc65c4d 2fdba142-40f8-4ae5-985b-c54d2f20231f /tidb/cdc/default/default/upstream/7365852461863404712 {"id":7365852461863404712,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2fdba142-40f8-4ae5-985b-c54d2f20231f {"id":"2fdba142-40f8-4ae5-985b-c54d2f20231f","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996184} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbcc65c4d 2fdba142-40f8-4ae5-985b-c54d2f20231f /tidb/cdc/default/default/upstream/7365852461863404712 {"id":7365852461863404712,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2fdba142-40f8-4ae5-985b-c54d2f20231f {"id":"2fdba142-40f8-4ae5-985b-c54d2f20231f","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996184} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbcc65c4d 2fdba142-40f8-4ae5-985b-c54d2f20231f /tidb/cdc/default/default/upstream/7365852461863404712 {"id":7365852461863404712,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.cli.2748.out cli changefeed create --start-ts=449575959013621761 '--sink-uri=file:///tmp/tidb_cdc_test/sink_retry/storage_test/ticdc-sink-retry-test-31426?protocol=canal-json&enable-tidb-extension=true' [Mon May 6 19:49:47 CST 2024] <<<<<< START cdc server in storage_csv_update case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.storage_csv_update.62906292.out server --log-file /tmp/tidb_cdc_test/storage_csv_update/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/storage_csv_update/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 29-th check, retry later Create changefeed successfully! ID: a8d5150c-d398-42d2-80c3-17db29055ca1 Info: {"upstream_id":7365852461863404712,"namespace":"default","id":"a8d5150c-d398-42d2-80c3-17db29055ca1","sink_uri":"file:///tmp/tidb_cdc_test/sink_retry/storage_test/ticdc-sink-retry-test-31426?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:49:47.627714283+08:00","start_ts":449575959013621761,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575959013621761,"checkpoint_ts":449575959013621761,"checkpoint_time":"2024-05-06 19:49:41.540"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [Mon May 6 19:49:47 CST 2024] <<<<<< START cdc server in processor_err_chan case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/ProcessorAddTableError=1*return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_err_chan.27612763.out server --log-file /tmp/tidb_cdc_test/processor_err_chan/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_err_chan/cdc_data --cluster-id default --config /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/processor_err_chan/conf/server.toml --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x + workdir=/tmp/tidb_cdc_test/row_format + sink_uri='file:///tmp/tidb_cdc_test/row_format/storage_test/ticdc-row-format-test-9545?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/row_format ++ date + echo '[Mon May 6 19:49:48 CST 2024] <<<<<< START storage consumer in row_format case >>>>>>' [Mon May 6 19:49:48 CST 2024] <<<<<< START storage consumer in row_format case >>>>>> + cd /tmp/tidb_cdc_test/row_format + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/row_format + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/row_format/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/row_format/storage_test/ticdc-row-format-test-9545?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' + set +x + workdir=/tmp/tidb_cdc_test/foreign_key + sink_uri='file:///tmp/tidb_cdc_test/foreign_key/storage_test/ticdc-foreign-key-test-20237?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/foreign_key ++ date + echo '[Mon May 6 19:49:48 CST 2024] <<<<<< START storage consumer in foreign_key case >>>>>>' [Mon May 6 19:49:48 CST 2024] <<<<<< START storage consumer in foreign_key case >>>>>> + cd /tmp/tidb_cdc_test/foreign_key + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/foreign_key + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/foreign_key/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/foreign_key/storage_test/ticdc-foreign-key-test-20237?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' + set +x + workdir=/tmp/tidb_cdc_test/sink_retry + sink_uri='file:///tmp/tidb_cdc_test/sink_retry/storage_test/ticdc-sink-retry-test-31426?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/sink_retry ++ date + echo '[Mon May 6 19:49:49 CST 2024] <<<<<< START storage consumer in sink_retry case >>>>>>' [Mon May 6 19:49:49 CST 2024] <<<<<< START storage consumer in sink_retry case >>>>>> + cd /tmp/tidb_cdc_test/sink_retry + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/sink_retry + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/sink_retry/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/sink_retry/storage_test/ticdc-sink-retry-test-31426?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/consistent_replicate_storage_s3/run.sh using Sink-Type: storage... <<================= * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:24927; Connection refused * Closing connection 0 You are running an older version of MinIO released 3 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:49:49 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fd8f3224-fcce-43a9-8ce8-9cdaa7a89459 {"id":"fd8f3224-fcce-43a9-8ce8-9cdaa7a89459","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996186} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbccdf633 fd8f3224-fcce-43a9-8ce8-9cdaa7a89459 /tidb/cdc/default/default/upstream/7365852472817576998 {"id":7365852472817576998,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fd8f3224-fcce-43a9-8ce8-9cdaa7a89459 {"id":"fd8f3224-fcce-43a9-8ce8-9cdaa7a89459","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996186} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbccdf633 fd8f3224-fcce-43a9-8ce8-9cdaa7a89459 /tidb/cdc/default/default/upstream/7365852472817576998 {"id":7365852472817576998,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fd8f3224-fcce-43a9-8ce8-9cdaa7a89459 {"id":"fd8f3224-fcce-43a9-8ce8-9cdaa7a89459","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996186} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbccdf633 fd8f3224-fcce-43a9-8ce8-9cdaa7a89459 /tidb/cdc/default/default/upstream/7365852472817576998 {"id":7365852472817576998,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + workdir=/tmp/tidb_cdc_test/gc_safepoint + sink_uri='file:///tmp/tidb_cdc_test/gc_safepoint/storage_test/ticdc-gc-safepoint-11556?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/gc_safepoint ++ date + echo '[Mon May 6 19:49:49 CST 2024] <<<<<< START storage consumer in gc_safepoint case >>>>>>' [Mon May 6 19:49:49 CST 2024] <<<<<< START storage consumer in gc_safepoint case >>>>>> + cd /tmp/tidb_cdc_test/gc_safepoint + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/gc_safepoint + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/gc_safepoint/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/gc_safepoint/storage_test/ticdc-gc-safepoint-11556?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' 0 Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > table ddl_manager.finish_mark not exists for 30-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc) 3667979264 bytes in 8.22 secs (445989440 bytes/sec) [Pipeline] { [Pipeline] cache < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:49:50 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1ebcc1a1-8131-492b-ae2e-5ebf40755584 {"id":"1ebcc1a1-8131-492b-ae2e-5ebf40755584","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996187} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbcbcf9c0 1ebcc1a1-8131-492b-ae2e-5ebf40755584 /tidb/cdc/default/default/upstream/7365852451682056360 {"id":7365852451682056360,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1ebcc1a1-8131-492b-ae2e-5ebf40755584 {"id":"1ebcc1a1-8131-492b-ae2e-5ebf40755584","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996187} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbcbcf9c0 1ebcc1a1-8131-492b-ae2e-5ebf40755584 /tidb/cdc/default/default/upstream/7365852451682056360 {"id":7365852451682056360,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1ebcc1a1-8131-492b-ae2e-5ebf40755584 {"id":"1ebcc1a1-8131-492b-ae2e-5ebf40755584","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996187} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbcbcf9c0 1ebcc1a1-8131-492b-ae2e-5ebf40755584 /tidb/cdc/default/default/upstream/7365852451682056360 {"id":7365852451682056360,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:49:50 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5450752d-df02-446e-93a2-b5e04c06c7f9 {"id":"5450752d-df02-446e-93a2-b5e04c06c7f9","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996187} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbcd9a32b 5450752d-df02-446e-93a2-b5e04c06c7f9 /tidb/cdc/default/default/upstream/7365852488377750902 {"id":7365852488377750902,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5450752d-df02-446e-93a2-b5e04c06c7f9 {"id":"5450752d-df02-446e-93a2-b5e04c06c7f9","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996187} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbcd9a32b 5450752d-df02-446e-93a2-b5e04c06c7f9 /tidb/cdc/default/default/upstream/7365852488377750902 {"id":7365852488377750902,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5450752d-df02-446e-93a2-b5e04c06c7f9 {"id":"5450752d-df02-446e-93a2-b5e04c06c7f9","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996187} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbcd9a32b 5450752d-df02-446e-93a2-b5e04c06c7f9 /tidb/cdc/default/default/upstream/7365852488377750902 {"id":7365852488377750902,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.storage_csv_update.cli.6345.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/storage_csv_update/storage_test?flush-interval=5s&enable-tidb-extension=true' --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_csv_update/conf/changefeed.toml * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:24927 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 17CCE3F2E5AF0002 < X-Xss-Protection: 1; mode=block < Date: Mon, 06 May 2024 11:49:50 GMT < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + workdir=/tmp/tidb_cdc_test/processor_err_chan + sink_uri='file:///tmp/tidb_cdc_test/processor_err_chan/storage_test/ticdc-processor-err-chan-13966?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/processor_err_chan ++ date + echo '[Mon May 6 19:49:50 CST 2024] <<<<<< START storage consumer in processor_err_chan case >>>>>>' [Mon May 6 19:49:50 CST 2024] <<<<<< START storage consumer in processor_err_chan case >>>>>> + cd /tmp/tidb_cdc_test/processor_err_chan + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/processor_err_chan + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/processor_err_chan/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/processor_err_chan/storage_test/ticdc-processor-err-chan-13966?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' check_changefeed_state http://127.0.0.1:2379 ed078d62-fae1-4854-ba4d-3cf99988f4f0 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=ed078d62-fae1-4854-ba4d-3cf99988f4f0 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c ed078d62-fae1-4854-ba4d-3cf99988f4f0 -s + info='{ "upstream_id": 7365852451682056360, "namespace": "default", "id": "ed078d62-fae1-4854-ba4d-3cf99988f4f0", "state": "normal", "checkpoint_tso": 449575961306595331, "checkpoint_time": "2024-05-06 19:49:50.287", "error": null }' + echo '{ "upstream_id": 7365852451682056360, "namespace": "default", "id": "ed078d62-fae1-4854-ba4d-3cf99988f4f0", "state": "normal", "checkpoint_tso": 449575961306595331, "checkpoint_time": "2024-05-06 19:49:50.287", "error": null }' { "upstream_id": 7365852451682056360, "namespace": "default", "id": "ed078d62-fae1-4854-ba4d-3cf99988f4f0", "state": "normal", "checkpoint_tso": 449575961306595331, "checkpoint_time": "2024-05-06 19:49:50.287", "error": null } ++ echo '{' '"upstream_id":' 7365852451682056360, '"namespace":' '"default",' '"id":' '"ed078d62-fae1-4854-ba4d-3cf99988f4f0",' '"state":' '"normal",' '"checkpoint_tso":' 449575961306595331, '"checkpoint_time":' '"2024-05-06' '19:49:50.287",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7365852451682056360, '"namespace":' '"default",' '"id":' '"ed078d62-fae1-4854-ba4d-3cf99988f4f0",' '"state":' '"normal",' '"checkpoint_tso":' 449575961306595331, '"checkpoint_time":' '"2024-05-06' '19:49:50.287",' '"error":' null '}' ++ jq -r .error.message Create changefeed successfully! ID: c05b195f-e81d-4cf8-b9a5-ec1ab43653f5 Info: {"upstream_id":7365852488377750902,"namespace":"default","id":"c05b195f-e81d-4cf8-b9a5-ec1ab43653f5","sink_uri":"file:///tmp/tidb_cdc_test/storage_csv_update/storage_test?flush-interval=5s\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:49:50.629953927+08:00","start_ts":449575961368461317,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575961368461317,"checkpoint_ts":449575961368461317,"checkpoint_time":"2024-05-06 19:49:50.523"} PASS Bucket 's3://logbucket/' created [Mon May 6 19:49:50 CST 2024] <<<<<< run test case consistent_replicate_storage_s3 success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Exiting on signal: INTERRUPT + message=null + [[ ! null =~ null ]] run task successfully coverage: 2.7% of statements in github.com/pingcap/tiflow/... check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 31-th check, retry later + set +x ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later check diff failed 2-th time, retry later table ddl_manager.finish_mark not exists for 32-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 3-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/consistent_partition_table/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:49:54 CST 2024] <<<<<< run test case consistent_partition_table success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 3-th time, retry later table ddl_manager.finish_mark not exists for 33-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f4bcf40012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-b8w74, pid:1654, start at 2024-05-06 19:49:55.162044144 +0800 CST m=+5.321783847 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:55.170 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:55.133 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:55.133 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 4-th time, retry later table foreign_key.finish_mark not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 34-th check, retry later check diff failed 4-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f4bcf40012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-b8w74, pid:1654, start at 2024-05-06 19:49:55.162044144 +0800 CST m=+5.321783847 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:55.170 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:55.133 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:55.133 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f4bcd80002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-b8w74, pid:1738, start at 2024-05-06 19:49:55.126565759 +0800 CST m=+5.200274151 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:55.133 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:55.126 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:55.126 +0800 All versions after safe point can be accessed. (DO NOT EDIT) check diff failed 5-th time, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_big_messages_v2/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:49:57 CST 2024] <<<<<< run test case kafka_big_messages_v2 success! >>>>>> Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cli_tls_with_auth/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cli_tls_with_auth/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/cli_tls_with_auth/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/cli_tls_with_auth/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cli_tls_with_auth/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table foreign_key.finish_mark not exists for 2-th check, retry later check diff failed 5-th time, retry later The 1 times to try to start tls tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/cli_tls_with_auth Starting TLS PD... table ddl_manager.finish_mark not exists for 35-th check, retry later Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 check diff successfully check_safepoint_forward http://127.0.0.1:2379 7365852472817576998 449575963560771589 449575961070141442 Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc) 3667979264 bytes in 8.01 secs (457814220 bytes/sec) [Pipeline] { [Pipeline] cache table foreign_key.finish_mark not exists for 3-th check, retry later run task successfully check_changefeed_state http://127.0.0.1:2379 171b9797-acc4-4e89-a2de-4e1b7a17de15 stopped null + endpoints=http://127.0.0.1:2379 + changefeed_id=171b9797-acc4-4e89-a2de-4e1b7a17de15 + expected_state=stopped + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 171b9797-acc4-4e89-a2de-4e1b7a17de15 -s check diff failed 6-th time, retry later + info='{ "upstream_id": 7365852472817576998, "namespace": "default", "id": "171b9797-acc4-4e89-a2de-4e1b7a17de15", "state": "stopped", "checkpoint_tso": 449575964085059590, "checkpoint_time": "2024-05-06 19:50:00.886", "error": null }' + echo '{ "upstream_id": 7365852472817576998, "namespace": "default", "id": "171b9797-acc4-4e89-a2de-4e1b7a17de15", "state": "stopped", "checkpoint_tso": 449575964085059590, "checkpoint_time": "2024-05-06 19:50:00.886", "error": null }' { "upstream_id": 7365852472817576998, "namespace": "default", "id": "171b9797-acc4-4e89-a2de-4e1b7a17de15", "state": "stopped", "checkpoint_tso": 449575964085059590, "checkpoint_time": "2024-05-06 19:50:00.886", "error": null } ++ echo '{' '"upstream_id":' 7365852472817576998, '"namespace":' '"default",' '"id":' '"171b9797-acc4-4e89-a2de-4e1b7a17de15",' '"state":' '"stopped",' '"checkpoint_tso":' 449575964085059590, '"checkpoint_time":' '"2024-05-06' '19:50:00.886",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7365852472817576998, '"namespace":' '"default",' '"id":' '"171b9797-acc4-4e89-a2de-4e1b7a17de15",' '"state":' '"stopped",' '"checkpoint_tso":' 449575964085059590, '"checkpoint_time":' '"2024-05-06' '19:50:00.886",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_equal http://127.0.0.1:2379 7365852472817576998 table ddl_manager.finish_mark not exists for 36-th check, retry later table foreign_key.finish_mark not exists for 4-th check, retry later table ddl_manager.finish_mark not exists for 37-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... Starting TLS TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/multi_tables_ddl_v2/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:50:04 CST 2024] <<<<<< run test case multi_tables_ddl_v2 success! >>>>>> run task successfully wait process cdc.test exit for 2-th time... check_changefeed_state http://127.0.0.1:2379 171b9797-acc4-4e89-a2de-4e1b7a17de15 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=171b9797-acc4-4e89-a2de-4e1b7a17de15 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 171b9797-acc4-4e89-a2de-4e1b7a17de15 -s + info='{ "upstream_id": 7365852472817576998, "namespace": "default", "id": "171b9797-acc4-4e89-a2de-4e1b7a17de15", "state": "normal", "checkpoint_tso": 449575964085059590, "checkpoint_time": "2024-05-06 19:50:00.886", "error": null }' + echo '{ "upstream_id": 7365852472817576998, "namespace": "default", "id": "171b9797-acc4-4e89-a2de-4e1b7a17de15", "state": "normal", "checkpoint_tso": 449575964085059590, "checkpoint_time": "2024-05-06 19:50:00.886", "error": null }' { "upstream_id": 7365852472817576998, "namespace": "default", "id": "171b9797-acc4-4e89-a2de-4e1b7a17de15", "state": "normal", "checkpoint_tso": 449575964085059590, "checkpoint_time": "2024-05-06 19:50:00.886", "error": null } ++ echo '{' '"upstream_id":' 7365852472817576998, '"namespace":' '"default",' '"id":' '"171b9797-acc4-4e89-a2de-4e1b7a17de15",' '"state":' '"normal",' '"checkpoint_tso":' 449575964085059590, '"checkpoint_time":' '"2024-05-06' '19:50:00.886",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7365852472817576998, '"namespace":' '"default",' '"id":' '"171b9797-acc4-4e89-a2de-4e1b7a17de15",' '"state":' '"normal",' '"checkpoint_tso":' 449575964085059590, '"checkpoint_time":' '"2024-05-06' '19:50:00.886",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_forward http://127.0.0.1:2379 7365852472817576998 449575964085059589 449575964085059590 table foreign_key.finish_mark not exists for 5-th check, retry later table ddl_manager.finish_mark not exists for 38-th check, retry later table row_format.finish_mark not exists for 1-th check, retry later table row_format.finish_mark not exists for 2-th check, retry later Starting TLS TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying TLS TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) run task successfully check_changefeed_state http://127.0.0.1:2379 171b9797-acc4-4e89-a2de-4e1b7a17de15 stopped null + endpoints=http://127.0.0.1:2379 + changefeed_id=171b9797-acc4-4e89-a2de-4e1b7a17de15 + expected_state=stopped + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 171b9797-acc4-4e89-a2de-4e1b7a17de15 -s wait process cdc.test exit for 3-th time... + info='{ "upstream_id": 7365852472817576998, "namespace": "default", "id": "171b9797-acc4-4e89-a2de-4e1b7a17de15", "state": "stopped", "checkpoint_tso": 449575964085059590, "checkpoint_time": "2024-05-06 19:50:00.886", "error": null }' + echo '{ "upstream_id": 7365852472817576998, "namespace": "default", "id": "171b9797-acc4-4e89-a2de-4e1b7a17de15", "state": "stopped", "checkpoint_tso": 449575964085059590, "checkpoint_time": "2024-05-06 19:50:00.886", "error": null }' { "upstream_id": 7365852472817576998, "namespace": "default", "id": "171b9797-acc4-4e89-a2de-4e1b7a17de15", "state": "stopped", "checkpoint_tso": 449575964085059590, "checkpoint_time": "2024-05-06 19:50:00.886", "error": null } ++ echo '{' '"upstream_id":' 7365852472817576998, '"namespace":' '"default",' '"id":' '"171b9797-acc4-4e89-a2de-4e1b7a17de15",' '"state":' '"stopped",' '"checkpoint_tso":' 449575964085059590, '"checkpoint_time":' '"2024-05-06' '19:50:00.886",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7365852472817576998, '"namespace":' '"default",' '"id":' '"171b9797-acc4-4e89-a2de-4e1b7a17de15",' '"state":' '"stopped",' '"checkpoint_tso":' 449575964085059590, '"checkpoint_time":' '"2024-05-06' '19:50:00.886",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:50:07 CST 2024] <<<<<< run test case processor_err_chan success! >>>>>> table row_format.finish_mark not exists for 3-th check, retry later check_changefeed_state http://127.0.0.1:2379 04df7828-34d7-4c10-8322-f1c11ca0617a normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=04df7828-34d7-4c10-8322-f1c11ca0617a + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 04df7828-34d7-4c10-8322-f1c11ca0617a -s table foreign_key.finish_mark not exists for 6-th check, retry later + info='{ "upstream_id": 7365852472817576998, "namespace": "default", "id": "04df7828-34d7-4c10-8322-f1c11ca0617a", "state": "normal", "checkpoint_tso": 449575965723459585, "checkpoint_time": "2024-05-06 19:50:07.136", "error": null }' + echo '{ "upstream_id": 7365852472817576998, "namespace": "default", "id": "04df7828-34d7-4c10-8322-f1c11ca0617a", "state": "normal", "checkpoint_tso": 449575965723459585, "checkpoint_time": "2024-05-06 19:50:07.136", "error": null }' { "upstream_id": 7365852472817576998, "namespace": "default", "id": "04df7828-34d7-4c10-8322-f1c11ca0617a", "state": "normal", "checkpoint_tso": 449575965723459585, "checkpoint_time": "2024-05-06 19:50:07.136", "error": null } ++ echo '{' '"upstream_id":' 7365852472817576998, '"namespace":' '"default",' '"id":' '"04df7828-34d7-4c10-8322-f1c11ca0617a",' '"state":' '"normal",' '"checkpoint_tso":' 449575965723459585, '"checkpoint_time":' '"2024-05-06' '19:50:07.136",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7365852472817576998, '"namespace":' '"default",' '"id":' '"04df7828-34d7-4c10-8322-f1c11ca0617a",' '"state":' '"normal",' '"checkpoint_tso":' 449575965723459585, '"checkpoint_time":' '"2024-05-06' '19:50:07.136",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_equal http://127.0.0.1:2379 7365852472817576998 table ddl_manager.finish_mark not exists for 39-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/multi_topics_v2/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:50:07 CST 2024] <<<<<< run test case multi_topics_v2 success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table foreign_key.finish_mark not exists for 7-th check, retry later table row_format.finish_mark not exists for 4-th check, retry later table ddl_manager.finish_mark not exists for 40-th check, retry later run task successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Changefeed remove successfully. ID: 171b9797-acc4-4e89-a2de-4e1b7a17de15 CheckpointTs: 449575964085059590 SinkURI: file:///tmp/tidb_cdc_test/gc_safepoint/storage_test/ticdc-gc-safepoint-11556?protocol=canal-json&enable-tidb-extension=true check_safepoint_forward http://127.0.0.1:2379 7365852472817576998 449575965723459584 449575965723459585 449575964085059590 table foreign_key.finish_mark not exists for 8-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc) 3667979264 bytes in 8.19 secs (447665219 bytes/sec) [Pipeline] { [Pipeline] cache table ddl_manager.finish_mark not exists for 41-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f5c2bc0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-b8w74, pid:3360, start at 2024-05-06 19:50:11.914273089 +0800 CST m=+5.321099165 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:11.924 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:11.887 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:11.887 +0800 All versions after safe point can be accessed. (DO NOT EDIT) + pd_host=127.0.0.1 + pd_port=2579 + is_tls=true + '[' true == true ']' ++ run_cdc_cli tso query --pd=https://127.0.0.1:2579 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.3425.out cli tso query --pd=https://127.0.0.1:2579 run task successfully table row_format.finish_mark not exists for 5-th check, retry later Changefeed remove successfully. ID: 04df7828-34d7-4c10-8322-f1c11ca0617a CheckpointTs: 449575966981488642 SinkURI: file:///tmp/tidb_cdc_test/gc_safepoint/storage_test/ticdc-gc-safepoint-11556?protocol=canal-json&enable-tidb-extension=true check_safepoint_cleared http://127.0.0.1:2379 7365852472817576998 run task successfully table foreign_key.finish_mark not exists for 9-th check, retry later table ddl_manager.finish_mark not exists for 42-th check, retry later wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... + set +x + tso='449575967283478529 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575967283478529 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:50:14 CST 2024] <<<<<< START cdc server in cli_tls_with_auth case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates ']' + GO_FAILPOINTS= + curl_status_cmd='curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.34753477.out server --log-file /tmp/tidb_cdc_test/cli_tls_with_auth/cdc_cli_tls_with_auth_tls1.log --log-level debug --data-dir /tmp/tidb_cdc_test/cli_tls_with_auth/cdc_data_cli_tls_with_auth_tls1 --cluster-id default --config /tmp/tidb_cdc_test/cli_tls_with_auth/server.toml --ca /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/server.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/server-key.pem --cert-allowed-cn client --addr 127.0.0.1:8300 --pd https://127.0.0.1:2579 + (( i = 0 )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 3-th time... table row_format.finish_mark not exists for 6-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:50:15 CST 2024] <<<<<< run test case gc_safepoint success! >>>>>> table foreign_key.finish_mark not exists for 10-th check, retry later table ddl_manager.finish_mark not exists for 43-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/run.sh using Sink-Type: storage... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup + source /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/storage_cleanup + CDC_BINARY=cdc.test + SINK_TYPE=storage + EXIST_FILES=() + CLEANED_FILES=() + trap stop_tidb_cluster EXIT + run storage + '[' storage '!=' storage ']' + rm -rf /tmp/tidb_cdc_test/storage_cleanup + mkdir -p /tmp/tidb_cdc_test/storage_cleanup + start_tidb_cluster --workdir /tmp/tidb_cdc_test/storage_cleanup The 1 times to try to start tidb cluster... table row_format.finish_mark not exists for 7-th check, retry later table foreign_key.finish_mark not exists for 11-th check, retry later table ddl_manager.finish_mark not exists for 44-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/ca.pem CApath: none * NSS: client certificate from file * subject: CN=client * start date: Feb 18 07:48:00 2020 GMT * expire date: Jan 25 07:48:00 2120 GMT * common name: client * issuer: CN=My own CA,O=PingCAP,L=Beijing,ST=Beijing,C=CN * SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 * Server certificate: * subject: CN=tidb-server * start date: Feb 18 09:11:00 2020 GMT * expire date: Jan 25 09:11:00 2120 GMT * common name: tidb-server * issuer: CN=My own CA,O=PingCAP,L=Beijing,ST=Beijing,C=CN * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:50:17 GMT < Content-Length: 1289 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3002f00e-57b2-43dd-bd81-32eae760f6e3 {"id":"3002f00e-57b2-43dd-bd81-32eae760f6e3","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996215} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d8f4dbd43514d 3002f00e-57b2-43dd-bd81-32eae760f6e3 /tidb/cdc/default/default/upstream/7365852607464303422 {"id":7365852607464303422,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3002f00e-57b2-43dd-bd81-32eae760f6e3 {"id":"3002f00e-57b2-43dd-bd81-32eae760f6e3","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996215} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d8f4dbd43514d 3002f00e-57b2-43dd-bd81-32eae760f6e3 /tidb/cdc/default/default/upstream/7365852607464303422 {"id":7365852607464303422,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3002f00e-57b2-43dd-bd81-32eae760f6e3 {"id":"3002f00e-57b2-43dd-bd81-32eae760f6e3","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996215} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d8f4dbd43514d 3002f00e-57b2-43dd-bd81-32eae760f6e3 /tidb/cdc/default/default/upstream/7365852607464303422 {"id":7365852607464303422,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'etcd info' + break + set +x check diff failed 1-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/storage_cleanup Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... table row_format.finish_mark not exists for 8-th check, retry later table foreign_key.finish_mark not exists for 12-th check, retry later table ddl_manager.finish_mark not exists for 45-th check, retry later + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.3541.out cli changefeed create --start-ts=449575967283478529 '--sink-uri=file:///tmp/tidb_cdc_test/cli_tls_with_auth/storage_test/ticdc-cli-test-4181?protocol=canal-json&enable-tidb-extension=true' --tz=Asia/Shanghai -c=custom-changefeed-name [WARN] --tz is deprecated in changefeed settings. Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7365852607464303422,"namespace":"default","id":"custom-changefeed-name","sink_uri":"file:///tmp/tidb_cdc_test/cli_tls_with_auth/storage_test/ticdc-cli-test-4181?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:50:20.583841142+08:00","start_ts":449575967283478529,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575967283478529,"checkpoint_ts":449575967283478529,"checkpoint_time":"2024-05-06 19:50:13.087"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... table sink_retry.finish_mark_1 exists check diff successfully ***************** properties ***************** "mysql.db"="sink_retry" "workload"="core" "mysql.port"="4000" "dotransactions"="false" "operationcount"="0" "insertproportion"="0" "threadcount"="2" "readproportion"="0" "mysql.host"="127.0.0.1" "mysql.user"="root" "updateproportion"="0" "scanproportion"="0" "requestdistribution"="uniform" "recordcount"="10" "readallfields"="true" ********************************************** Run finished, takes 5.362374ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2540.4, Avg(us): 913, Min(us): 590, Max(us): 2085, 95th(us): 3000, 99th(us): 3000 check diff failed 2-th time, retry later table row_format.finish_mark not exists for 9-th check, retry later table foreign_key.finish_mark not exists for 13-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/cli_tls_with_auth + sink_uri='file:///tmp/tidb_cdc_test/cli_tls_with_auth/storage_test/ticdc-cli-test-4181?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/cli_tls_with_auth ++ date + echo '[Mon May 6 19:50:22 CST 2024] <<<<<< START storage consumer in cli_tls_with_auth case >>>>>>' [Mon May 6 19:50:22 CST 2024] <<<<<< START storage consumer in cli_tls_with_auth case >>>>>> + cd /tmp/tidb_cdc_test/cli_tls_with_auth + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/cli_tls_with_auth + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/cli_tls_with_auth/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/cli_tls_with_auth/storage_test/ticdc-cli-test-4181?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table test.simple not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 46-th check, retry later check diff failed 3-th time, retry later table row_format.finish_mark not exists for 10-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table foreign_key.finish_mark not exists for 14-th check, retry later table ddl_manager.finish_mark not exists for 47-th check, retry later ls /tmp/tidb_cdc_test/storage_csv_update/storage_test/test/test_update 449575961774784550 run task successfully start check update + workdir=/tmp/tidb_cdc_test/storage_csv_update + sink_uri='file:///tmp/tidb_cdc_test/storage_csv_update/storage_test?flush-interval=5s&enable-tidb-extension=true' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_csv_update/conf/changefeed.toml + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/storage_csv_update ++ date + echo '[Mon May 6 19:50:24 CST 2024] <<<<<< START storage consumer in storage_csv_update case >>>>>>' [Mon May 6 19:50:24 CST 2024] <<<<<< START storage consumer in storage_csv_update case >>>>>> + cd /tmp/tidb_cdc_test/storage_csv_update + '[' /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_csv_update/conf/changefeed.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/storage_csv_update + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/storage_csv_update/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/storage_csv_update/storage_test?flush-interval=5s&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_csv_update/conf/changefeed.toml table test.simple not exists for 2-th check, retry later check diff failed 4-th time, retry later table row_format.finish_mark not exists for 11-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/changefeed_reconstruct/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table foreign_key.finish_mark not exists for 15-th check, retry later table ddl_manager.finish_mark not exists for 48-th check, retry later table test.simple not exists for 3-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/changefeed_reconstruct Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... table row_format.finish_mark not exists for 12-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully table foreign_key.finish_mark not exists for 16-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/changefeed_pause_resume/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... wait process cdc.test exit for 1-th time... table ddl_manager.finish_mark not exists for 49-th check, retry later wait process cdc.test exit for 2-th time... table test.simple not exists for 4-th check, retry later wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:50:29 CST 2024] <<<<<< run test case kv_client_stream_reconnect success! >>>>>> table row_format.finish_mark not exists for 13-th check, retry later table foreign_key.finish_mark exists check diff successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... table test.simple not exists for 5-th check, retry later table ddl_manager.finish_mark not exists for 50-th check, retry later wait process cdc.test exit for 2-th time... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc) 3667979264 bytes in 14.76 secs (248532837 bytes/sec) [Pipeline] { [Pipeline] cache wait process cdc.test exit for 3-th time... table row_format.finish_mark not exists for 14-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f6e8600018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-vs88z, pid:2306, start at 2024-05-06 19:50:30.719218109 +0800 CST m=+5.599423827 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:30.729 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:30.730 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:30.730 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:50:31 CST 2024] <<<<<< run test case foreign_key success! >>>>>> table ddl_manager.finish_mark not exists for 51-th check, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table row_format.finish_mark not exists for 15-th check, retry later table test.simple not exists for 6-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f6e8600018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-vs88z, pid:2306, start at 2024-05-06 19:50:30.719218109 +0800 CST m=+5.599423827 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:30.729 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:30.730 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:30.730 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f6eb6c0009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-vs88z, pid:2383, start at 2024-05-06 19:50:30.884032907 +0800 CST m=+5.708949577 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:30.895 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:30.875 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:30.875 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/storage_cleanup/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/storage_cleanup/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/storage_cleanup/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/storage_cleanup/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/storage_cleanup/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table ddl_manager.finish_mark not exists for 52-th check, retry later check diff failed 1-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/changefeed_pause_resume Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... check diff failed 2-th time, retry later table test.simple exists table test.`simple-dash` exists + endpoints=https://127.0.0.1:2579 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c custom-changefeed-name -s + info='{ "upstream_id": 7365852607464303422, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449575972670537733, "checkpoint_time": "2024-05-06 19:50:33.637", "error": null }' + echo '{ "upstream_id": 7365852607464303422, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449575972670537733, "checkpoint_time": "2024-05-06 19:50:33.637", "error": null }' { "upstream_id": 7365852607464303422, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449575972670537733, "checkpoint_time": "2024-05-06 19:50:33.637", "error": null } ++ echo '{' '"upstream_id":' 7365852607464303422, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449575972670537733, '"checkpoint_time":' '"2024-05-06' '19:50:33.637",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7365852607464303422, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449575972670537733, '"checkpoint_time":' '"2024-05-06' '19:50:33.637",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] changefeed count 1 check pass, pd_addr: https://127.0.0.1:2579 table row_format.finish_mark not exists for 16-th check, retry later table ddl_manager.finish_mark not exists for 53-th check, retry later + cd /tmp/tidb_cdc_test/storage_cleanup + run_cdc_server --workdir /tmp/tidb_cdc_test/storage_cleanup --binary cdc.test [Mon May 6 19:50:35 CST 2024] <<<<<< START cdc server in storage_cleanup case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS= + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.storage_cleanup.36443646.out server --log-file /tmp/tidb_cdc_test/storage_cleanup/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/storage_cleanup/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Error: [CDC:ErrChangefeedUpdateRefused]changefeed update error: can only update changefeed config when it is stopped or failed update changefeed config should fail when changefeed is running, got Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config SyncPointInterval] From: To:0xc001621b00} {Type:update Path:[Config SyncPointRetention] From: To:0xc001621b08} {Type:update Path:[Config Consistent] From: To:0xc001293e30} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.3814.out cli changefeed --changefeed-id custom-changefeed-name pause Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully [Mon May 6 19:50:36 CST 2024] <<<<<< run test case storage_csv_update success! >>>>>> table row_format.finish_mark not exists for 17-th check, retry later + set +x table ddl_manager.finish_mark not exists for 54-th check, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:50:38 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/89c40f29-53f7-48d6-9424-c40f2c3a00d1 {"id":"89c40f29-53f7-48d6-9424-c40f2c3a00d1","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996235} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd8c9035 89c40f29-53f7-48d6-9424-c40f2c3a00d1 /tidb/cdc/default/default/upstream/7365852687622286969 {"id":7365852687622286969,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/89c40f29-53f7-48d6-9424-c40f2c3a00d1 {"id":"89c40f29-53f7-48d6-9424-c40f2c3a00d1","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996235} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd8c9035 89c40f29-53f7-48d6-9424-c40f2c3a00d1 /tidb/cdc/default/default/upstream/7365852687622286969 {"id":7365852687622286969,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/89c40f29-53f7-48d6-9424-c40f2c3a00d1 {"id":"89c40f29-53f7-48d6-9424-c40f2c3a00d1","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996235} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd8c9035 89c40f29-53f7-48d6-9424-c40f2c3a00d1 /tidb/cdc/default/default/upstream/7365852687622286969 {"id":7365852687622286969,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + generate_historic_files + local target_bucket=storage_test ++ date -d yesterday +%Y-%m-%d + yesterday=2024-05-05 ++ date -d '2 days ago' +%Y-%m-%d + day_before_yesterday=2024-05-04 + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test multi_data_type 2024-05-05 10 false + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test + local table=multi_data_type + local day=2024-05-05 + local file_cnt=10 + local should_clean=false + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05 ++ seq 1 10 + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/1.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/2.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/3.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/4.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/5.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/6.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/7.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/8.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/9.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/10.data + '[' false == true ']' + EXIST_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test multi_charset 2024-05-04 10 true + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test + local table=multi_charset + local day=2024-05-04 + local file_cnt=10 + local should_clean=true + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04 ++ seq 1 10 + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/1.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/2.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/3.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/4.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/5.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/6.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/7.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/8.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/9.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/10.data + '[' true == true ']' + CLEANED_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test binary_columns 2024-05-04 10 true + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test + local table=binary_columns + local day=2024-05-04 + local file_cnt=10 + local should_clean=true + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04 ++ seq 1 10 + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/1.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/2.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/3.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/4.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/5.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/6.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/7.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/8.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/9.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/10.data + '[' true == true ']' + CLEANED_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test multi_data_type_dummy 2024-05-04 10 true + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test + local table=multi_data_type_dummy + local day=2024-05-04 + local file_cnt=10 + local should_clean=true + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04 ++ seq 1 10 + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/1.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/2.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/3.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/4.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/5.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/6.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/7.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/8.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/9.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/10.data + '[' true == true ']' + CLEANED_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test multi_charset_dummy 2024-05-04 10 true + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test + local table=multi_charset_dummy + local day=2024-05-04 + local file_cnt=10 + local should_clean=true + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04 ++ seq 1 10 + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/1.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/2.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/3.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/4.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/5.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/6.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/7.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/8.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/9.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/10.data + '[' true == true ']' + CLEANED_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test binary_columns_dummy 2024-05-05 10 false + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test + local table=binary_columns_dummy + local day=2024-05-05 + local file_cnt=10 + local should_clean=false + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05 ++ seq 1 10 + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/1.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/2.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/3.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/4.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/5.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/6.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/7.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/8.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/9.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/10.data + '[' false == true ']' + EXIST_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test2 multi_data_type 2024-05-04 10 true + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test2 + local table=multi_data_type + local day=2024-05-04 + local file_cnt=10 + local should_clean=true + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04 ++ seq 1 10 + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/1.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/2.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/3.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/4.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/5.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/6.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/7.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/8.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/9.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/10.data + '[' true == true ']' + CLEANED_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test2 multi_charset 2024-05-04 10 true + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test2 + local table=multi_charset + local day=2024-05-04 + local file_cnt=10 + local should_clean=true + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04 ++ seq 1 10 + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/1.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/2.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/3.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/4.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/5.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/6.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/7.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/8.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/9.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/10.data + '[' true == true ']' + CLEANED_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test2 binary_columns 2024-05-05 10 false + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test2 + local table=binary_columns + local day=2024-05-05 + local file_cnt=10 + local should_clean=false + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05 ++ seq 1 10 + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/1.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/2.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/3.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/4.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/5.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/6.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/7.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/8.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/9.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/10.data + '[' false == true ']' + EXIST_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test_default test multi_data_type 2022-01-01 10 false + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test_default + local schema=test + local table=multi_data_type + local day=2022-01-01 + local file_cnt=10 + local should_clean=false + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01 ++ seq 1 10 + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/1.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/2.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/3.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/4.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/5.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/6.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/7.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/8.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/9.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/10.data + '[' false == true ']' + EXIST_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test_default test multi_charset 2022-01-02 10 false + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test_default + local schema=test + local table=multi_charset + local day=2022-01-02 + local file_cnt=10 + local should_clean=false + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02 ++ seq 1 10 + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/1.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/2.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/3.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/4.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/5.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/6.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/7.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/8.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/9.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/10.data + '[' false == true ']' + EXIST_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test_default test binary_columns 2022-01-03 10 false + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test_default + local schema=test + local table=binary_columns + local day=2022-01-03 + local file_cnt=10 + local should_clean=false + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03 ++ seq 1 10 + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/1.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/2.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/3.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/4.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/5.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/6.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/7.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/8.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/9.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/10.data + '[' false == true ']' + EXIST_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/meta/CDC.index + SINK_URI_DEFAULT='file:///tmp/tidb_cdc_test/storage_cleanup/storage_test_default?flush-interval=5s' + run_cdc_cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/storage_cleanup/storage_test_default?flush-interval=5s' -c default-config-test --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/changefeed-default.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.storage_cleanup.cli.3867.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/storage_cleanup/storage_test_default?flush-interval=5s' -c default-config-test --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/changefeed-default.toml Create changefeed successfully! ID: default-config-test Info: {"upstream_id":7365852687622286969,"namespace":"default","id":"default-config-test","sink_uri":"file:///tmp/tidb_cdc_test/storage_cleanup/storage_test_default?flush-interval=5s","create_time":"2024-05-06T19:50:39.139829857+08:00","start_ts":449575974084280325,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575974084280325,"checkpoint_ts":449575974084280325,"checkpoint_time":"2024-05-06 19:50:39.030"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... table row_format.finish_mark not exists for 18-th check, retry later table ddl_manager.finish_mark not exists for 55-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f75e480016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l, pid:4206, start at 2024-05-06 19:50:38.269338772 +0800 CST m=+6.036605228 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:38.278 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:38.276 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:38.276 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f75e480016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l, pid:4206, start at 2024-05-06 19:50:38.269338772 +0800 CST m=+6.036605228 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:38.278 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:38.276 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:38.276 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f758cc0009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l, pid:4282, start at 2024-05-06 19:50:37.884883757 +0800 CST m=+5.570416052 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:37.891 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:37.875 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:37.875 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + endpoints=https://127.0.0.1:2579 + changefeed_id=custom-changefeed-name + expected_state=stopped + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c custom-changefeed-name -s + info='{ "upstream_id": 7365852607464303422, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 449575973195087879, "checkpoint_time": "2024-05-06 19:50:35.638", "error": null }' + echo '{ "upstream_id": 7365852607464303422, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 449575973195087879, "checkpoint_time": "2024-05-06 19:50:35.638", "error": null }' { "upstream_id": 7365852607464303422, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 449575973195087879, "checkpoint_time": "2024-05-06 19:50:35.638", "error": null } ++ echo '{' '"upstream_id":' 7365852607464303422, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 449575973195087879, '"checkpoint_time":' '"2024-05-06' '19:50:35.638",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7365852607464303422, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 449575973195087879, '"checkpoint_time":' '"2024-05-06' '19:50:35.638",' '"error":' null '}' ++ jq -r .error.message ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.3902.out cli changefeed update --pd=https://127.0.0.1:2579 --config=/tmp/tidb_cdc_test/cli_tls_with_auth/changefeed.toml --no-confirm --changefeed-id custom-changefeed-name =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/cdc/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:50:40 CST 2024] <<<<<< run test case cdc success! >>>>>> table row_format.finish_mark not exists for 19-th check, retry later Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config SyncPointInterval] From: To:0xc00150a888} {Type:update Path:[Config SyncPointRetention] From: To:0xc00150a898} {Type:update Path:[Config Consistent] From: To:0xc001524d20} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} Update changefeed config successfully! ID: custom-changefeed-name Info: {"upstream_id":7365852607464303422,"namespace":"default","id":"custom-changefeed-name","sink_uri":"file:///tmp/tidb_cdc_test/cli_tls_with_auth/storage_test/ticdc-cli-test-4181?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:50:20.583841142+08:00","start_ts":449575967283478529,"admin_job_type":1,"config":{"memory_quota":1073741824,"case_sensitive":true,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_table_monitor":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"delete_only_output_handle_key_columns":null,"content_compatible":null,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"stopped","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":0,"checkpoint_ts":449575973195087879,"checkpoint_time":"2024-05-06 19:50:35.638"} PASS coverage: 3.0% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 56-th check, retry later + set +x + sleep 20 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Mon May 6 19:50:42 CST 2024] <<<<<< START cdc server in changefeed_reconstruct case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS= + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_reconstruct.55995601.out server --log-file /tmp/tidb_cdc_test/changefeed_reconstruct/cdcserver1.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_reconstruct/cdc_dataserver1 --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + grep -q 'failed to get info:' + echo '' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table row_format.finish_mark not exists for 20-th check, retry later + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.3939.out cli changefeed --changefeed-id custom-changefeed-name resume =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/split_region/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table ddl_manager.finish_mark not exists for 57-th check, retry later PASS coverage: 2.3% of statements in github.com/pingcap/tiflow/... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f7a964001d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz, pid:4632, start at 2024-05-06 19:50:43.074230225 +0800 CST m=+5.400449216 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:43.084 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:43.084 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:43.084 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f7a964001d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz, pid:4632, start at 2024-05-06 19:50:43.074230225 +0800 CST m=+5.400449216 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:43.084 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:43.084 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:43.084 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:50:45 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d8eca3c8-a83b-4274-91c7-4f84450001d7 {"id":"d8eca3c8-a83b-4274-91c7-4f84450001d7","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996242} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbdaa3b31 d8eca3c8-a83b-4274-91c7-4f84450001d7 /tidb/cdc/default/default/upstream/7365852720192660454 {"id":7365852720192660454,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d8eca3c8-a83b-4274-91c7-4f84450001d7 {"id":"d8eca3c8-a83b-4274-91c7-4f84450001d7","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996242} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbdaa3b31 d8eca3c8-a83b-4274-91c7-4f84450001d7 /tidb/cdc/default/default/upstream/7365852720192660454 {"id":7365852720192660454,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d8eca3c8-a83b-4274-91c7-4f84450001d7 {"id":"d8eca3c8-a83b-4274-91c7-4f84450001d7","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996242} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbdaa3b31 d8eca3c8-a83b-4274-91c7-4f84450001d7 /tidb/cdc/default/default/upstream/7365852720192660454 {"id":7365852720192660454,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table row_format.finish_mark not exists for 21-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/changefeed_reconstruct + sink_uri='file:///tmp/tidb_cdc_test/changefeed_reconstruct/storage_test/ticdc-changefeed-reconstruct-328?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/changefeed_reconstruct ++ date + echo '[Mon May 6 19:50:45 CST 2024] <<<<<< START storage consumer in changefeed_reconstruct case >>>>>>' [Mon May 6 19:50:45 CST 2024] <<<<<< START storage consumer in changefeed_reconstruct case >>>>>> + cd /tmp/tidb_cdc_test/changefeed_reconstruct + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/changefeed_reconstruct + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/changefeed_reconstruct/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/changefeed_reconstruct/storage_test/ticdc-changefeed-reconstruct-328?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table ddl_manager.finish_mark not exists for 58-th check, retry later ***************** properties ***************** "updateproportion"="0" "threadcount"="4" "readallfields"="true" "mysql.port"="4000" "mysql.db"="changefeed_reconstruct" "readproportion"="0" "operationcount"="0" "requestdistribution"="uniform" "workload"="core" "mysql.user"="root" "mysql.host"="127.0.0.1" "insertproportion"="0" "recordcount"="50" "scanproportion"="0" "dotransactions"="false" ********************************************** Run finished, takes 18.596089ms INSERT - Takes(s): 0.0, Count: 48, OPS: 3311.2, Avg(us): 1463, Min(us): 966, Max(us): 4469, 95th(us): 5000, 99th(us): 5000 table changefeed_reconstruct.usertable not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f7bffc0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz, pid:4710, start at 2024-05-06 19:50:44.505014599 +0800 CST m=+6.768088553 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:44.515 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:44.479 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:44.479 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table row_format.finish_mark not exists for 22-th check, retry later table ddl_manager.finish_mark not exists for 59-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/split_region Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc) 3667979264 bytes in 15.58 secs (235440019 bytes/sec) [Pipeline] { [Pipeline] cache table changefeed_reconstruct.usertable not exists for 2-th check, retry later [Mon May 6 19:50:48 CST 2024] <<<<<< START cdc server in changefeed_pause_resume case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_pause_resume.59655967.out server --log-file /tmp/tidb_cdc_test/changefeed_pause_resume/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_pause_resume/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + endpoints=https://127.0.0.1:2579 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c custom-changefeed-name -s + info='{ "upstream_id": 7365852607464303422, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449575976353660930, "checkpoint_time": "2024-05-06 19:50:47.687", "error": null }' + echo '{ "upstream_id": 7365852607464303422, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449575976353660930, "checkpoint_time": "2024-05-06 19:50:47.687", "error": null }' { "upstream_id": 7365852607464303422, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449575976353660930, "checkpoint_time": "2024-05-06 19:50:47.687", "error": null } ++ echo '{' '"upstream_id":' 7365852607464303422, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449575976353660930, '"checkpoint_time":' '"2024-05-06' '19:50:47.687",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7365852607464303422, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449575976353660930, '"checkpoint_time":' '"2024-05-06' '19:50:47.687",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.4021.out cli changefeed --changefeed-id custom-changefeed-name remove Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table row_format.finish_mark not exists for 23-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/ddl_puller_lag/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... Changefeed remove successfully. ID: custom-changefeed-name CheckpointTs: 449575976615804930 SinkURI: file:///tmp/tidb_cdc_test/cli_tls_with_auth/storage_test/ticdc-cli-test-4181?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 60-th check, retry later table changefeed_reconstruct.usertable not exists for 3-th check, retry later Starting Upstream TiDB... + set +x table row_format.finish_mark exists check diff successfully Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:50:51 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fbca44c5-6eed-4219-8a44-8f651c48815b {"id":"fbca44c5-6eed-4219-8a44-8f651c48815b","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996249} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbdc37334 fbca44c5-6eed-4219-8a44-8f651c48815b /tidb/cdc/default/default/upstream/7365852739816017819 {"id":7365852739816017819,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fbca44c5-6eed-4219-8a44-8f651c48815b {"id":"fbca44c5-6eed-4219-8a44-8f651c48815b","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996249} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbdc37334 fbca44c5-6eed-4219-8a44-8f651c48815b /tidb/cdc/default/default/upstream/7365852739816017819 {"id":7365852739816017819,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fbca44c5-6eed-4219-8a44-8f651c48815b {"id":"fbca44c5-6eed-4219-8a44-8f651c48815b","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996249} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbdc37334 fbca44c5-6eed-4219-8a44-8f651c48815b /tidb/cdc/default/default/upstream/7365852739816017819 {"id":7365852739816017819,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + workdir=/tmp/tidb_cdc_test/changefeed_pause_resume + sink_uri='file:///tmp/tidb_cdc_test/changefeed_pause_resume/storage_test/ticdc-changefeed-pause-resume-12307?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/changefeed_pause_resume ++ date + echo '[Mon May 6 19:50:51 CST 2024] <<<<<< START storage consumer in changefeed_pause_resume case >>>>>>' [Mon May 6 19:50:51 CST 2024] <<<<<< START storage consumer in changefeed_pause_resume case >>>>>> + cd /tmp/tidb_cdc_test/changefeed_pause_resume + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/changefeed_pause_resume + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/changefeed_pause_resume/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/changefeed_pause_resume/storage_test/ticdc-changefeed-pause-resume-12307?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' wait process cdc.test exit for 1-th time... table ddl_manager.finish_mark not exists for 61-th check, retry later table changefeed_reconstruct.usertable not exists for 4-th check, retry later table changefeed_pause_resume.t1 not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... table sink_retry.finish_mark_2 exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:50:52 CST 2024] <<<<<< run test case sink_retry success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:50:53 CST 2024] <<<<<< run test case row_format success! >>>>>> start tidb cluster in /tmp/tidb_cdc_test/ddl_puller_lag Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... table ddl_manager.finish_mark not exists for 62-th check, retry later table changefeed_reconstruct.usertable not exists for 5-th check, retry later table changefeed_pause_resume.t1 not exists for 2-th check, retry later changefeed count 0 check pass, pd_addr: https://127.0.0.1:2579 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.4091.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/cli_tls_with_auth/storage_test/ticdc-cli-test-4181?protocol=canal-json&enable-tidb-extension=true' --tz=Asia/Shanghai -c=custom-changefeed-name \033[0;36m<<< Run all test success >>>\033[0m [WARN] --tz is deprecated in changefeed settings. Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7365852607464303422,"namespace":"default","id":"custom-changefeed-name","sink_uri":"file:///tmp/tidb_cdc_test/cli_tls_with_auth/storage_test/ticdc-cli-test-4181?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:50:55.133932266+08:00","start_ts":449575978254467074,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575978254467074,"checkpoint_ts":449575978254467074,"checkpoint_time":"2024-05-06 19:50:54.938"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... Verifying downstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table ddl_manager.finish_mark not exists for 63-th check, retry later table changefeed_reconstruct.usertable exists check diff successfully + set +x table changefeed_pause_resume.t1 not exists for 3-th check, retry later wait process 5604 exit for 1-th time... wait process 5604 exit for 2-th time... wait process 5604 exit for 3-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f880140013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w9f11, pid:4335, start at 2024-05-06 19:50:56.796253827 +0800 CST m=+5.474566302 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:56.806 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:56.773 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:56.773 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... table ddl_manager.finish_mark not exists for 64-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (5604) - No such process wait process 5604 exit for 4-th time... process 5604 already exit check_no_capture http://127.0.0.1:2379 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Mon May 6 19:50:58 CST 2024] <<<<<< START cdc server in changefeed_reconstruct case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_reconstruct.58865888.out server --log-file /tmp/tidb_cdc_test/changefeed_reconstruct/cdcserver2.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_reconstruct/cdc_dataserver2 --cluster-id default --addr 127.0.0.1:8300 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table changefeed_pause_resume.t1 not exists for 4-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f880140013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w9f11, pid:4335, start at 2024-05-06 19:50:56.796253827 +0800 CST m=+5.474566302 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:56.806 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:56.773 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:56.773 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f882680013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w9f11, pid:4419, start at 2024-05-06 19:50:56.950809118 +0800 CST m=+5.573707957 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:56.960 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:56.922 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:56.922 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/split_region/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/split_region/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/split_region/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/split_region/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/split_region/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + endpoints=https://127.0.0.1:2579 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c custom-changefeed-name -s + info='{ "upstream_id": 7365852607464303422, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449575979237244931, "checkpoint_time": "2024-05-06 19:50:58.687", "error": null }' + echo '{ "upstream_id": 7365852607464303422, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449575979237244931, "checkpoint_time": "2024-05-06 19:50:58.687", "error": null }' { "upstream_id": 7365852607464303422, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449575979237244931, "checkpoint_time": "2024-05-06 19:50:58.687", "error": null } ++ echo '{' '"upstream_id":' 7365852607464303422, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449575979237244931, '"checkpoint_time":' '"2024-05-06' '19:50:58.687",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7365852607464303422, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449575979237244931, '"checkpoint_time":' '"2024-05-06' '19:50:58.687",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] table ddl_manager.finish_mark not exists for 65-th check, retry later table changefeed_pause_resume.t1 not exists for 5-th check, retry later + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.4219.out cli unsafe delete-service-gc-safepoint Confirm that you know what this command will do and use it at your own risk [Y/N] CDC service GC safepoint truncated in PD! PASS ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) coverage: 2.2% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:51:01 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/4215d5e1-8bd0-465d-a8a6-37352fbdb85a {UpstreamID:7365852720192660454 Namespace:default ID:4215d5e1-8bd0-465d-a8a6-37352fbdb85a SinkURI:file:///tmp/tidb_cdc_test/changefeed_reconstruct/storage_test/ticdc-changefeed-reconstruct-328?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:50:45.774659656 +0800 CST StartTs:449575975826489349 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003a3d440 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575975839596552} {CheckpointTs:449575978356178945 MinTableBarrierTs:449575979928780802 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449575978356178945, checkpointTs: 449575978356178945, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a1f5aeb5-a3c3-4625-93a5-e03655ececc6 {"id":"a1f5aeb5-a3c3-4625-93a5-e03655ececc6","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996258} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbdaa3bed a1f5aeb5-a3c3-4625-93a5-e03655ececc6 /tidb/cdc/default/default/changefeed/info/4215d5e1-8bd0-465d-a8a6-37352fbdb85a {"upstream-id":7365852720192660454,"namespace":"default","changefeed-id":"4215d5e1-8bd0-465d-a8a6-37352fbdb85a","sink-uri":"file:///tmp/tidb_cdc_test/changefeed_reconstruct/storage_test/ticdc-changefeed-reconstruct-328?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:50:45.774659656+08:00","start-ts":449575975826489349,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575975839596552} /tidb/cdc/default/default/changefeed/status/4215d5e1-8bd0-465d-a8a6-37352fbdb85a {"checkpoint-ts":449575978356178945,"min-table-barrier-ts":449575979928780802,"admin-job-type":0} /tidb/cdc/default/default/task/position/a1f5aeb5-a3c3-4625-93a5-e03655ececc6/4215d5e1-8bd0-465d-a8a6-37352fbdb85a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852720192660454 {"id":7365852720192660454,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/4215d5e1-8bd0-465d-a8a6-37352fbdb85a {UpstreamID:7365852720192660454 Namespace:default ID:4215d5e1-8bd0-465d-a8a6-37352fbdb85a SinkURI:file:///tmp/tidb_cdc_test/changefeed_reconstruct/storage_test/ticdc-changefeed-reconstruct-328?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:50:45.774659656 +0800 CST StartTs:449575975826489349 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003a3d440 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575975839596552} {CheckpointTs:449575978356178945 MinTableBarrierTs:449575979928780802 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449575978356178945, checkpointTs: 449575978356178945, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a1f5aeb5-a3c3-4625-93a5-e03655ececc6 {"id":"a1f5aeb5-a3c3-4625-93a5-e03655ececc6","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996258} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbdaa3bed a1f5aeb5-a3c3-4625-93a5-e03655ececc6 /tidb/cdc/default/default/changefeed/info/4215d5e1-8bd0-465d-a8a6-37352fbdb85a {"upstream-id":7365852720192660454,"namespace":"default","changefeed-id":"4215d5e1-8bd0-465d-a8a6-37352fbdb85a","sink-uri":"file:///tmp/tidb_cdc_test/changefeed_reconstruct/storage_test/ticdc-changefeed-reconstruct-328?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:50:45.774659656+08:00","start-ts":449575975826489349,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575975839596552} /tidb/cdc/default/default/changefeed/status/4215d5e1-8bd0-465d-a8a6-37352fbdb85a {"checkpoint-ts":449575978356178945,"min-table-barrier-ts":449575979928780802,"admin-job-type":0} /tidb/cdc/default/default/task/position/a1f5aeb5-a3c3-4625-93a5-e03655ececc6/4215d5e1-8bd0-465d-a8a6-37352fbdb85a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"er+ grep -q 'failed to get info:' ror":null,"warning":null} /tidb/cdc/default/default/upstream/7365852720192660454 {"id":7365852720192660454,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/4215d5e1-8bd0-465d-a8a6-37352fbdb85a {UpstreamID:7365852720192660454 Namespace:default ID:4215d5e1-8bd0-465d-a8a6-37352fbdb85a SinkURI:file:///tmp/tidb_cdc_test/changefeed_reconstruct/storage_test/ticdc-changefeed-reconstruct-328?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:50:45.774659656 +0800 CST StartTs:449575975826489349 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003a3d440 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575975839596552} {CheckpointTs:449575978356178945 MinTableBarrierTs:449575979928780802 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449575978356178945, checkpointTs: 449575978356178945, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a1f5aeb5-a3c3-4625-93a5-e03655ececc6 {"id":"a1f5aeb5-a3c3-4625-93a5-e03655ececc6","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996258} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbdaa3bed a1f5aeb5-a3c3-4625-93a5-e03655ececc6 /tidb/cdc/default/default/changefeed/info/4215d5e1-8bd0-465d-a8a6-37352fbdb85a {"upstream-id":7365852720192660454,"namespace":"default","changefeed-id":"4215d5e1-8bd0-465d-a8a6-37352fbdb85a","sink-uri":"file:///tmp/tidb_cdc_test/changefeed_reconstruct/storage_test/ticdc-changefeed-reconstruct-328?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:50:45.774659656+08:00","start-ts":449575975826489349,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575975839596552} /tidb/cdc/default/default/changefeed/status/4215d5e1-8bd0-465d-a8a6-37352fbdb85a {"checkpoint-ts":449575978356178945,"min-table-barrier-ts":449575979928780802,"admin-job-type":0} /tidb/cdc/default/default/task/position/a1f5aeb5-a3c3-4625-93a5-e03655ececc6/4215d5e1-8bd0-465d-a8a6-37352fbdb85a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"er+ grep -q 'etcd info' ror":null,"warning":null} /tidb/cdc/default/default/upstream/7365852720192660454 {"id":7365852720192660454,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list --pd=http://127.0.0.1:2379 2>&1 | grep id + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.cli.5719.out cli tso query --pd=http://127.0.0.1:2379 table ddl_manager.finish_mark not exists for 66-th check, retry later table changefeed_pause_resume.t1 exists table changefeed_pause_resume.t2 exists table changefeed_pause_resume.t3 exists + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.4252.out cli unsafe reset --no-confirm --pd=https://127.0.0.1:2579 "id": "a1f5aeb5-a3c3-4625-93a5-e03655ececc6", "cluster-id": "default" run task successfully reset and all metadata truncated in PD! PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + tso='449575980098912257 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575980098912257 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:51:03 CST 2024] <<<<<< START cdc server in split_region case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.57635765.out server --log-file /tmp/tidb_cdc_test/split_region/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/split_region/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + check_file_exists true + local all_should_exist=true + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/1.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/2.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/3.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/4.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/5.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/6.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/7.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/8.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/9.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/10.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/1.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/2.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/3.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/4.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/5.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/6.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/7.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/8.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/9.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/10.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/1.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/2.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/3.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/4.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/5.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/6.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/7.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/8.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/9.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/10.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/1.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/2.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/3.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/4.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/5.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/6.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/7.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/8.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/9.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/10.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/1.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/2.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/3.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/4.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/5.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/6.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/7.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/8.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/9.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/10.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/1.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/2.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/3.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/4.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/5.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/6.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/7.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/8.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/9.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/10.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/1.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/2.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/3.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/4.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/5.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/6.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/7.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/8.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/9.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/10.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/1.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/2.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/3.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/4.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/5.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/6.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/7.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/8.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/9.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/10.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/1.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/2.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/3.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/4.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/5.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/6.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/7.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/8.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/9.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/10.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/1.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/2.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/3.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/4.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/5.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/6.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/7.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/8.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/9.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/10.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/1.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/2.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/3.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/4.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/5.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/6.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/7.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/8.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/9.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/10.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/1.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/2.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/3.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/4.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/5.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/6.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/7.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/8.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/9.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/10.data ']' + SINK_URI='file:///tmp/tidb_cdc_test/storage_cleanup/storage_test?flush-interval=5s' + run_cdc_cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/storage_cleanup/storage_test?flush-interval=5s' --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.storage_cleanup.cli.3946.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/storage_cleanup/storage_test?flush-interval=5s' --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/changefeed.toml Create changefeed successfully! ID: 19d94c6a-1229-471a-95f0-45d86969a345 Info: {"upstream_id":7365852687622286969,"namespace":"default","id":"19d94c6a-1229-471a-95f0-45d86969a345","sink_uri":"file:///tmp/tidb_cdc_test/storage_cleanup/storage_test?flush-interval=5s","create_time":"2024-05-06T19:51:02.239763763+08:00","start_ts":449575980152913923,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"cloud_storage_config":{"file_expiration_days":1,"file_cleanup_cron_spec":"* * * * * *"},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575980152913923,"checkpoint_ts":449575980152913923,"checkpoint_time":"2024-05-06 19:51:02.180"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... + set +x + sleep 20 table ddl_manager.finish_mark not exists for 67-th check, retry later check diff failed 1-th time, retry later capture_id: a1f5aeb5-a3c3-4625-93a5-e03655ececc6 check_processor_table_count http://127.0.0.1:2379 4215d5e1-8bd0-465d-a8a6-37352fbdb85a a1f5aeb5-a3c3-4625-93a5-e03655ececc6 1 run task successfully check_processor_table_count http://127.0.0.1:2379 4215d5e1-8bd0-465d-a8a6-37352fbdb85a a1f5aeb5-a3c3-4625-93a5-e03655ececc6 0 + set +x VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f8f2800016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-nrrkt, pid:3933, start at 2024-05-06 19:51:04.108926315 +0800 CST m=+6.212757555 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:04.116 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:04.096 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:04.096 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f8f2800016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-nrrkt, pid:3933, start at 2024-05-06 19:51:04.108926315 +0800 CST m=+6.212757555 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:04.116 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:04.096 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:04.096 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f8e6c00014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-nrrkt, pid:4013, start at 2024-05-06 19:51:03.387949599 +0800 CST m=+5.436805759 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:03.394 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:03.395 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:03.395 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table count 1 does equal to expected count 0 run task failed 1-th time, retry later table ddl_manager.finish_mark not exists for 68-th check, retry later check diff failed 2-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:51:06 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0d210048-b0ab-41a1-a671-4ad6c5a43fad {"id":"0d210048-b0ab-41a1-a671-4ad6c5a43fad","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996264} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbdfab258 0d210048-b0ab-41a1-a671-4ad6c5a43fad /tidb/cdc/default/default/upstream/7365852798364713785 {"id":7365852798364713785,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0d210048-b0ab-41a1-a671-4ad6c5a43fad {"id":"0d210048-b0ab-41a1-a671-4ad6c5a43fad","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996264} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbdfab258 0d210048-b0ab-41a1-a671-4ad6c5a43fad /tidb/cdc/default/default/upstream/7365852798364713785 {"id":7365852798364713785,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0d210048-b0ab-41a1-a671-4ad6c5a43fad {"id":"0d210048-b0ab-41a1-a671-4ad6c5a43fad","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996264} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbdfab258 0d210048-b0ab-41a1-a671-4ad6c5a43fad /tidb/cdc/default/default/upstream/7365852798364713785 {"id":7365852798364713785,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.cli.5820.out cli changefeed create --start-ts=449575980098912257 '--sink-uri=file:///tmp/tidb_cdc_test/split_region/storage_test/ticdc-split-region-test-27658?protocol=canal-json&enable-tidb-extension=true' --config /tmp/tidb_cdc_test/split_region/pulsar_test.toml Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc) 3667979264 bytes in 16.35 secs (224327046 bytes/sec) [Pipeline] { [Pipeline] cache Create changefeed successfully! ID: e0cfcc18-e16a-433e-8765-8431b932ee25 Info: {"upstream_id":7365852798364713785,"namespace":"default","id":"e0cfcc18-e16a-433e-8765-8431b932ee25","sink_uri":"file:///tmp/tidb_cdc_test/split_region/storage_test/ticdc-split-region-test-27658?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:51:07.329919278+08:00","start_ts":449575980098912257,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575980098912257,"checkpoint_ts":449575980098912257,"checkpoint_time":"2024-05-06 19:51:01.974"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.cli.5240.out cli tso query --pd=http://127.0.0.1:2379 check_processor_table_count http://127.0.0.1:2379 4215d5e1-8bd0-465d-a8a6-37352fbdb85a a1f5aeb5-a3c3-4625-93a5-e03655ececc6 0 table ddl_manager.finish_mark not exists for 69-th check, retry later + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.4358.out cli unsafe resolve-lock --region=44 PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/drop_many_tables/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... check diff failed 3-th time, retry later + set +x + workdir=/tmp/tidb_cdc_test/split_region + sink_uri='file:///tmp/tidb_cdc_test/split_region/storage_test/ticdc-split-region-test-27658?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/split_region ++ date + echo '[Mon May 6 19:51:08 CST 2024] <<<<<< START storage consumer in split_region case >>>>>>' [Mon May 6 19:51:08 CST 2024] <<<<<< START storage consumer in split_region case >>>>>> + cd /tmp/tidb_cdc_test/split_region + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/split_region + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/split_region/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/split_region/storage_test/ticdc-split-region-test-27658?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table split_region.test1 not exists for 1-th check, retry later + set +x + tso='449575981598900225 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575981598900225 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x run task successfully ***************** properties ***************** "requestdistribution"="uniform" "mysql.db"="changefeed_reconstruct" "insertproportion"="0" "workload"="core" "dotransactions"="false" "mysql.user"="root" "readallfields"="true" "readproportion"="0" "updateproportion"="0" "recordcount"="50" "operationcount"="0" "scanproportion"="0" "mysql.port"="4000" "threadcount"="4" "mysql.host"="127.0.0.1" ********************************************** Run finished, takes 17.647057ms INSERT - Takes(s): 0.0, Count: 48, OPS: 3439.9, Avg(us): 1352, Min(us): 864, Max(us): 4103, 95th(us): 4000, 99th(us): 5000 table changefeed_reconstruct.usertable not exists for 1-th check, retry later + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.4393.out cli unsafe resolve-lock --region=44 --ts=449575980849430530 PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 70-th check, retry later [Mon May 6 19:51:10 CST 2024] <<<<<< START cdc server in ddl_puller_lag case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorDDLResolved=1*sleep(180000)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.52905292.out server --log-file /tmp/tidb_cdc_test/ddl_puller_lag/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_puller_lag/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 4-th time, retry later table split_region.test1 not exists for 2-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/changefeed_error/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table changefeed_reconstruct.usertable not exists for 2-th check, retry later + set +x % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 218 100 212 100 6 1661 47 --:--:-- --:--:-- --:--:-- 1669 { "error_msg": "[CDC:ErrAPIInvalidParam]invalid log level: json: cannot unmarshal string into Go value of type struct { Level string \"json:\\\"log_level\\\"\" }", "error_code": "CDC:ErrAPIInvalidParam" table ddl_manager.finish_mark not exists for 71-th check, retry later check diff failed 5-th time, retry later table split_region.test1 not exists for 3-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:51:13 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9655fc0c-d568-4257-b91a-8bed8ac83472 {"id":"9655fc0c-d568-4257-b91a-8bed8ac83472","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996270} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe126c50 9655fc0c-d568-4257-b91a-8bed8ac83472 /tidb/cdc/default/default/upstream/7365852826217048967 {"id":7365852826217048967,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9655fc0c-d568-4257-b91a-8bed8ac83472 {"id":"9655fc0c-d568-4257-b91a-8bed8ac83472","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996270} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe126c50 9655fc0c-d568-4257-b91a-8bed8ac83472 /tidb/cdc/default/default/upstream/7365852826217048967 {"id":7365852826217048967,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9655fc0c-d568-4257-b91a-8bed8ac83472 {"id":"9655fc0c-d568-4257-b91a-8bed8ac83472","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996270} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe126c50 9655fc0c-d568-4257-b91a-8bed8ac83472 /tidb/cdc/default/default/upstream/7365852826217048967 {"id":7365852826217048967,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.cli.5351.out cli changefeed create --start-ts=449575981598900225 '--sink-uri=file:///tmp/tidb_cdc_test/ddl_puller_lag/storage_test/ticdc-ddl-puller-lag-test-12026?protocol=canal-json&enable-tidb-extension=true' start tidb cluster in /tmp/tidb_cdc_test/changefeed_error Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... table changefeed_reconstruct.usertable not exists for 3-th check, retry later Create changefeed successfully! ID: 66d54ab5-a0ce-4980-9b2b-e7033eaa3317 Info: {"upstream_id":7365852826217048967,"namespace":"default","id":"66d54ab5-a0ce-4980-9b2b-e7033eaa3317","sink_uri":"file:///tmp/tidb_cdc_test/ddl_puller_lag/storage_test/ticdc-ddl-puller-lag-test-12026?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:51:13.757584128+08:00","start_ts":449575981598900225,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575981598900225,"checkpoint_ts":449575981598900225,"checkpoint_time":"2024-05-06 19:51:07.696"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 72-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/drop_many_tables Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... check diff failed 6-th time, retry later } % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 181 100 181 0 0 1376 0 --:--:-- --:--:-- --:--:-- 1381 { "version": "v8.1.0-alpha-23-ge1a27adc6", "git_hash": "e1a27adc69acc9d50db0c3fea2945bbd6d983e30", "id": "d2b792bb-907c-49ae-92de-64ca0b2b94de", "pid": 3480, "is_owner": true table split_region.test1 not exists for 4-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/ddl_puller_lag + sink_uri='file:///tmp/tidb_cdc_test/ddl_puller_lag/storage_test/ticdc-ddl-puller-lag-test-12026?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/ddl_puller_lag ++ date + echo '[Mon May 6 19:51:15 CST 2024] <<<<<< START storage consumer in ddl_puller_lag case >>>>>>' [Mon May 6 19:51:15 CST 2024] <<<<<< START storage consumer in ddl_puller_lag case >>>>>> + cd /tmp/tidb_cdc_test/ddl_puller_lag + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/ddl_puller_lag + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/ddl_puller_lag/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/ddl_puller_lag/storage_test/ticdc-ddl-puller-lag-test-12026?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' }wait process cdc.test exit for 1-th time... Verifying downstream PD is started... table changefeed_reconstruct.usertable not exists for 4-th check, retry later Verifying downstream PD is started... wait process cdc.test exit for 2-th time... table ddl_manager.finish_mark not exists for 73-th check, retry later wait process cdc.test exit for 3-th time... check diff failed 7-th time, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:51:16 CST 2024] <<<<<< run test case cli_tls_with_auth success! >>>>>> table split_region.test1 not exists for 5-th check, retry later Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table changefeed_reconstruct.usertable exists check diff successfully Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table ddl_manager.finish_mark not exists for 74-th check, retry later wait process cdc.test exit for 1-th time... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... table split_region.test1 not exists for 6-th check, retry later check diff failed 8-th time, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:51:19 CST 2024] <<<<<< run test case changefeed_reconstruct success! >>>>>> Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 75-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc) 3667979264 bytes in 10.85 secs (337969094 bytes/sec) [Pipeline] { [Pipeline] sh table split_region.test1 exists table split_region.test2 exists [Pipeline] sh check diff successfully + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G14 Run cases: changefeed_finish force_replicate_table PROW_JOB_ID=8fa694ce-ccab-4faa-bb8b-f95ef2091797 JENKINS_NODE_COOKIE=5fda818f-719d-4e83-8fca-b8ad0ec336aa BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/ GOLANG_VERSION=1.21.6 HOSTNAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-7rfgh HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test","buildid":"1787445299229233156","prowjobid":"8fa694ce-ccab-4faa-bb8b-f95ef2091797","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#61 TEST_GROUP=G14 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233156 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-tcn64 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-81bc5d443e840859e7b2b4bd4bb8be00446de8a572b18ada90bce15670edaaa5 NODE_LABELS=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-tcn64 p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-7rfgh GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-7rfgh GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=61 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/changefeed_finish/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh check diff failed 9-th time, retry later + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G17 Run cases: clustered_index processor_resolved_ts_fallback PROW_JOB_ID=8fa694ce-ccab-4faa-bb8b-f95ef2091797 JENKINS_NODE_COOKIE=3848d4aa-6019-4a66-a6d4-048bd18ea3bd BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/ GOLANG_VERSION=1.21.6 HOSTNAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rwfr3 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test","buildid":"1787445299229233156","prowjobid":"8fa694ce-ccab-4faa-bb8b-f95ef2091797","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#61 TEST_GROUP=G17 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233156 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-6884q GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-81bc5d443e840859e7b2b4bd4bb8be00446de8a572b18ada90bce15670edaaa5 NODE_LABELS=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rwfr3 p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-6884q GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rwfr3 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=61 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/clustered_index/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:51:21 CST 2024] <<<<<< skip test case clustered_index for storage! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/processor_resolved_ts_fallback/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G13 Run cases: tiflash region_merge common_1 PROW_JOB_ID=8fa694ce-ccab-4faa-bb8b-f95ef2091797 JENKINS_NODE_COOKIE=3d23b75d-f385-45c5-bed3-ab72615ce68e BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/ GOLANG_VERSION=1.21.6 HOSTNAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-dscp5 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test","buildid":"1787445299229233156","prowjobid":"8fa694ce-ccab-4faa-bb8b-f95ef2091797","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#61 TEST_GROUP=G13 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233156 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-98v6t GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-81bc5d443e840859e7b2b4bd4bb8be00446de8a572b18ada90bce15670edaaa5 NODE_LABELS=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-98v6t p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-dscp5 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-dscp5 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=61 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/tiflash/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G12 Run cases: many_pk_or_uk capture_session_done_during_task ddl_attributes PROW_JOB_ID=8fa694ce-ccab-4faa-bb8b-f95ef2091797 JENKINS_NODE_COOKIE=a6470127-ca6c-4a20-9ca4-70cae79570ed BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/ GOLANG_VERSION=1.21.6 HOSTNAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w7kft HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test","buildid":"1787445299229233156","prowjobid":"8fa694ce-ccab-4faa-bb8b-f95ef2091797","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#61 TEST_GROUP=G12 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233156 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-0pn5r GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-81bc5d443e840859e7b2b4bd4bb8be00446de8a572b18ada90bce15670edaaa5 NODE_LABELS=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-0pn5r p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w7kft GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w7kft GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=61 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/many_pk_or_uk/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:51:22 CST 2024] <<<<<< run test case many_pk_or_uk success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] sh table ddl_manager.finish_mark not exists for 76-th check, retry later + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G11 [Pipeline] sh Run cases: resolve_lock move_table autorandom generate_column PROW_JOB_ID=8fa694ce-ccab-4faa-bb8b-f95ef2091797 JENKINS_NODE_COOKIE=a81f0146-550f-4ffa-bcd9-260ff894f557 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/ GOLANG_VERSION=1.21.6 HOSTNAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-pst3l HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test","buildid":"1787445299229233156","prowjobid":"8fa694ce-ccab-4faa-bb8b-f95ef2091797","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#61 TEST_GROUP=G11 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233156 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-nwnml GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-81bc5d443e840859e7b2b4bd4bb8be00446de8a572b18ada90bce15670edaaa5 NODE_LABELS=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-pst3l p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-nwnml GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-pst3l GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=61 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/resolve_lock/run.sh using Sink-Type: storage... <<================= + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G15 Run cases: new_ci_collation batch_add_table multi_rocks PROW_JOB_ID=8fa694ce-ccab-4faa-bb8b-f95ef2091797 JENKINS_NODE_COOKIE=3d870295-fbe3-4476-a772-cc24dd39f0b2 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/ GOLANG_VERSION=1.21.6 HOSTNAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-njdk7 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test","buildid":"1787445299229233156","prowjobid":"8fa694ce-ccab-4faa-bb8b-f95ef2091797","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#61 TEST_GROUP=G15 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233156 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-v7nm2 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-81bc5d443e840859e7b2b4bd4bb8be00446de8a572b18ada90bce15670edaaa5 NODE_LABELS=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-v7nm2 p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-njdk7 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-njdk7 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=61 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/new_ci_collation/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:51:22 CST 2024] <<<<<< run test case new_ci_collation success! >>>>>> [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G16 Run cases: owner_resign processor_etcd_worker_delay sink_hang PROW_JOB_ID=8fa694ce-ccab-4faa-bb8b-f95ef2091797 JENKINS_NODE_COOKIE=9905849a-763e-4ee1-956f-97708aff862d BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/ GOLANG_VERSION=1.21.6 HOSTNAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-81vjm HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test","buildid":"1787445299229233156","prowjobid":"8fa694ce-ccab-4faa-bb8b-f95ef2091797","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#61 TEST_GROUP=G16 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233156 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-r4b17 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-81bc5d443e840859e7b2b4bd4bb8be00446de8a572b18ada90bce15670edaaa5 NODE_LABELS=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-81vjm p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-r4b17 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-81vjm GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=61 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/owner_resign/run.sh using Sink-Type: storage... <<================= check diff successfully The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G10 Run cases: default_value simple cdc_server_tips event_filter sql_mode PROW_JOB_ID=8fa694ce-ccab-4faa-bb8b-f95ef2091797 JENKINS_NODE_COOKIE=6e4af3bd-1bdf-4cb5-9b89-bb61f849f172 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/ GOLANG_VERSION=1.21.6 HOSTNAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test","buildid":"1787445299229233156","prowjobid":"8fa694ce-ccab-4faa-bb8b-f95ef2091797","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#61 TEST_GROUP=G10 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233156 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/61/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-f0r2j GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-81bc5d443e840859e7b2b4bd4bb8be00446de8a572b18ada90bce15670edaaa5 NODE_LABELS=p_tiflow_release-8_1_pull_cdc_integration_storage_test_61-f0r2j p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=61 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/default_value/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:51:23 CST 2024] <<<<<< run test case default_value success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 77-th check, retry later check diff failed 1-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/tiflash Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/changefeed_finish Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/processor_resolved_ts_fallback Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... + check_file_exists false + local all_should_exist=false + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/1.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/2.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/3.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/4.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/5.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/6.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/7.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/8.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/9.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-05-05/10.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/1.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/2.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/3.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/4.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/5.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/6.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/7.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/8.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/9.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-05-05/10.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/1.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/2.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/3.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/4.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/5.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/6.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/7.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/8.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/9.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-05-05/10.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/1.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/2.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/3.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/4.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/5.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/6.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/7.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/8.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/9.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/10.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/1.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/2.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/3.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/4.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/5.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/6.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/7.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/8.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/9.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/10.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/1.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/2.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/3.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/4.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/5.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/6.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/7.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/8.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/9.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/10.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/1.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/2.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/3.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/4.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/5.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/6.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/7.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/8.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/9.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-05-04/10.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/1.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/2.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/3.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/4.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/5.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/6.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/7.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/8.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/9.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-05-04/10.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/1.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/2.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/3.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/4.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/5.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/6.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/7.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/8.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/9.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-05-04/10.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/1.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/2.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/3.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/4.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/5.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/6.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/7.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/8.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/9.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-05-04/10.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/1.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/2.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/3.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/4.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/5.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/6.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/7.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/8.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/9.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-05-04/10.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/1.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/2.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/3.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/4.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/5.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/6.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/7.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/8.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/9.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-05-04/10.data ']' + run_sql_file /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/data/schema.sql 127.0.0.1 4000 + run_sql_file /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/data/data.sql 127.0.0.1 4000 + run_storage_consumer /tmp/tidb_cdc_test/storage_cleanup 'file:///tmp/tidb_cdc_test/storage_cleanup/storage_test?flush-interval=5s' /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/changefeed.toml '' + workdir=/tmp/tidb_cdc_test/storage_cleanup + sink_uri='file:///tmp/tidb_cdc_test/storage_cleanup/storage_test?flush-interval=5s' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/changefeed.toml + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/storage_cleanup ++ date + echo '[Mon May 6 19:51:24 CST 2024] <<<<<< START storage consumer in storage_cleanup case >>>>>>' [Mon May 6 19:51:24 CST 2024] <<<<<< START storage consumer in storage_cleanup case >>>>>> + cd /tmp/tidb_cdc_test/storage_cleanup + '[' /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/changefeed.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/storage_cleanup + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/storage_cleanup/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/storage_cleanup/storage_test?flush-interval=5s' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/changefeed.toml + sleep 8 table ddl_manager.finish_mark not exists for 78-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/batch_add_table/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/capture_session_done_during_task/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa40480014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-t7zm9, pid:4008, start at 2024-05-06 19:51:25.505937373 +0800 CST m=+5.515808912 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:25.512 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:25.509 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:25.509 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later table ddl_manager.finish_mark not exists for 79-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa37c80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-drnnn, pid:3954, start at 2024-05-06 19:51:24.951679783 +0800 CST m=+5.317944246 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:24.961 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:24.964 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:24.964 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa37c80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-drnnn, pid:3954, start at 2024-05-06 19:51:24.951679783 +0800 CST m=+5.317944246 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:24.961 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:24.964 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:24.964 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa393c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-drnnn, pid:4032, start at 2024-05-06 19:51:25.018515402 +0800 CST m=+5.299529055 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:25.024 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:25.007 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:25.007 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_error/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_error/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } The 1 times to try to start tidb cluster... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa40480014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-t7zm9, pid:4008, start at 2024-05-06 19:51:25.505937373 +0800 CST m=+5.515808912 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:25.512 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:25.509 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:25.509 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa41ec0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-t7zm9, pid:4072, start at 2024-05-06 19:51:25.610357936 +0800 CST m=+5.561785717 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:25.621 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:25.614 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:25.614 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/drop_many_tables/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/drop_many_tables/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release start tidb cluster in /tmp/tidb_cdc_test/capture_session_done_during_task Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_simple_basic/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:51:28 CST 2024] <<<<<< run test case kafka_simple_basic success! >>>>>> start tidb cluster in /tmp/tidb_cdc_test/batch_add_table Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... check diff failed 3-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5339.out cli tso query --pd=http://127.0.0.1:2379 Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.cli.5305.out cli tso query --pd=http://127.0.0.1:2379 table ddl_manager.finish_mark not exists for 80-th check, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 4-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/owner_resign Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... + set +x + tso='449575987528335361 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575987528335361 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + set +x + tso='449575987631620097 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575987631620097 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:51:32 CST 2024] <<<<<< START cdc server in drop_many_tables case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.53475349.out server --log-file /tmp/tidb_cdc_test/drop_many_tables/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/drop_many_tables/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 start tidb cluster in /tmp/tidb_cdc_test/resolve_lock Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... table ddl_manager.finish_mark not exists for 81-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + check_sync_diff /tmp/tidb_cdc_test/storage_cleanup /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/diff_config.toml 100 check diff failed 1-th time, retry later Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ***************** properties ***************** "readallfields"="true" "workload"="core" "insertproportion"="0" "recordcount"="20" "operationcount"="0" "mysql.host"="127.0.0.1" "mysql.port"="4000" "threadcount"="4" "dotransactions"="false" "requestdistribution"="uniform" "scanproportion"="0" "mysql.user"="root" "readproportion"="0" "updateproportion"="0" "mysql.db"="changefeed_error" ********************************************** Run finished, takes 11.090277ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3452.1, Avg(us): 2101, Min(us): 971, Max(us): 5293, 95th(us): 6000, 99th(us): 6000 [Mon May 6 19:51:32 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/NewChangefeedNoRetryError=1*return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.54025404.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 1-th time, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/simple/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff failed 5-th time, retry later Verifying downstream PD is started... table ddl_manager.finish_mark not exists for 82-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/multi_capture/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:51:35 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/46207851-a6b6-47ee-b7bd-000558b69c7c {"id":"46207851-a6b6-47ee-b7bd-000558b69c7c","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996292} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe64b03d 46207851-a6b6-47ee-b7bd-000558b69c7c /tidb/cdc/default/default/upstream/7365852912520916751 {"id":7365852912520916751,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/46207851-a6b6-47ee-b7bd-000558b69c7c {"id":"46207851-a6b6-47ee-b7bd-000558b69c7c","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996292} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe64b03d 46207851-a6b6-47ee-b7bd-000558b69c7c /tidb/cdc/default/default/upstream/7365852912520916751 {"id":7365852912520916751,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/46207851-a6b6-47ee-b7bd-000558b69c7c {"id":"46207851-a6b6-47ee-b7bd-000558b69c7c","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996292} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe64b03d 46207851-a6b6-47ee-b7bd-000558b69c7c /tidb/cdc/default/default/upstream/7365852912520916751 {"id":7365852912520916751,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.cli.5402.out cli changefeed create --start-ts=449575987631620097 '--sink-uri=file:///tmp/tidb_cdc_test/drop_many_tables/storage_test/ticdc-drop-tables-test-22462?protocol=canal-json&enable-tidb-extension=true' ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Create changefeed successfully! ID: b6998d9d-32d1-46fb-b283-efe9fe0978c0 Info: {"upstream_id":7365852912520916751,"namespace":"default","id":"b6998d9d-32d1-46fb-b283-efe9fe0978c0","sink_uri":"file:///tmp/tidb_cdc_test/drop_many_tables/storage_test/ticdc-drop-tables-test-22462?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:51:35.710801476+08:00","start_ts":449575987631620097,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575987631620097,"checkpoint_ts":449575987631620097,"checkpoint_time":"2024-05-06 19:51:30.709"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... check diff failed 6-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/simple Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... table ddl_manager.finish_mark not exists for 83-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 1 -eq 50 ']' + sleep 3 Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully + check_logs /tmp/tidb_cdc_test/storage_cleanup ++ date + echo '[Mon May 6 19:51:36 CST 2024] <<<<<< run test case storage_cleanup success! >>>>>>' [Mon May 6 19:51:36 CST 2024] <<<<<< run test case storage_cleanup success! >>>>>> + stop_tidb_cluster Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fadebc0002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-dscp5, pid:1186, start at 2024-05-06 19:51:35.600166598 +0800 CST m=+5.261888602 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:35.609 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:35.599 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:35.599 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 3-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_simple_basic_avro/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:51:36 CST 2024] <<<<<< run test case kafka_simple_basic_avro success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + workdir=/tmp/tidb_cdc_test/drop_many_tables + sink_uri='file:///tmp/tidb_cdc_test/drop_many_tables/storage_test/ticdc-drop-tables-test-22462?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/drop_many_tables ++ date + echo '[Mon May 6 19:51:37 CST 2024] <<<<<< START storage consumer in drop_many_tables case >>>>>>' [Mon May 6 19:51:37 CST 2024] <<<<<< START storage consumer in drop_many_tables case >>>>>> + cd /tmp/tidb_cdc_test/drop_many_tables + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/drop_many_tables + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/drop_many_tables/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/drop_many_tables/storage_test/ticdc-drop-tables-test-22462?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc already exists) check diff failed 7-th time, retry later table ddl_manager.finish_mark not exists for 84-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table drop_tables.c not exists for 1-th check, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fae2680002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rwfr3, pid:1248, start at 2024-05-06 19:51:35.83538494 +0800 CST m=+5.442362348 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:35.843 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:35.834 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:35.834 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fae2680002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rwfr3, pid:1248, start at 2024-05-06 19:51:35.83538494 +0800 CST m=+5.442362348 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:35.843 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:35.834 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:35.834 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fae4740014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rwfr3, pid:1313, start at 2024-05-06 19:51:36.009541375 +0800 CST m=+5.521563727 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:36.019 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:36.016 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:36.016 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fadebc0002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-dscp5, pid:1186, start at 2024-05-06 19:51:35.600166598 +0800 CST m=+5.261888602 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:35.609 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:35.599 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:35.599 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fadf780013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-dscp5, pid:1269, start at 2024-05-06 19:51:35.675128343 +0800 CST m=+5.284754454 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:35.684 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:35.646 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:35.646 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/tiflash/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/tiflash/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 4-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fae8180016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-7rfgh, pid:1245, start at 2024-05-06 19:51:36.247732183 +0800 CST m=+5.255722230 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:36.254 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:36.249 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:36.249 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fae8180016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-7rfgh, pid:1245, start at 2024-05-06 19:51:36.247732183 +0800 CST m=+5.255722230 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:36.254 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:36.249 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:36.249 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36faea5c000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-7rfgh, pid:1300, start at 2024-05-06 19:51:36.352358082 +0800 CST m=+5.293096366 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:36.360 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:36.343 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:36.343 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_finish/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_finish/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:51:39 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c4c334b5-9074-4a15-9a6e-abf2a6371f87 {"id":"c4c334b5-9074-4a15-9a6e-abf2a6371f87","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996296} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe616464 c4c334b5-9074-4a15-9a6e-abf2a6371f87 /tidb/cdc/default/default/upstream/7365852910125063100 {"id":7365852910125063100,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c4c334b5-9074-4a15-9a6e-abf2a6371f87 {"id":"c4c334b5-9074-4a15-9a6e-abf2a6371f87","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996296} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe616464 c4c334b5-9074-4a15-9a6e-abf2a6371f87 /tidb/cdc/default/default/upstream/7365852910125063100 {"id":7365852910125063100,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c4c334b5-9074-4a15-9a6e-abf2a6371f87 {"id":"c4c334b5-9074-4a15-9a6e-abf2a6371f87","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996296} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe616464 c4c334b5-9074-4a15-9a6e-abf2a6371f87 /tidb/cdc/default/default/upstream/7365852910125063100 {"id":7365852910125063100,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5470.out cli changefeed create --start-ts=449575987528335361 '--sink-uri=file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-32148?protocol=canal-json&enable-tidb-extension=true' -c changefeed-error Create changefeed successfully! ID: changefeed-error Info: {"upstream_id":7365852910125063100,"namespace":"default","id":"changefeed-error","sink_uri":"file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-32148?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:51:39.698726487+08:00","start_ts":449575987528335361,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575987528335361,"checkpoint_ts":449575987528335361,"checkpoint_time":"2024-05-06 19:51:30.315"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table drop_tables.c not exists for 2-th check, retry later check diff failed 8-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 85-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_simple_handle_key_only/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:51:39 CST 2024] <<<<<< run test case kafka_simple_handle_key_only success! >>>>>> + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tiflash.cli.2636.out cli tso query --pd=http://127.0.0.1:2379 start tidb cluster in /tmp/tidb_cdc_test/multi_capture Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... [Mon May 6 19:51:40 CST 2024] <<<<<< START cdc server in processor_resolved_ts_fallback case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkExecDMLError=return(true);github.com/pingcap/tiflow/cdc/sink/dmlsink/mq/dmlproducer/KafkaSinkAsyncSendError=return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.25572559.out server --log-file /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fb2a6c0010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w7kft, pid:1256, start at 2024-05-06 19:51:40.461424189 +0800 CST m=+5.485676438 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:40.469 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:40.443 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:40.443 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully + set +x + workdir=/tmp/tidb_cdc_test/changefeed_error + sink_uri='file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-32148?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/changefeed_error ++ date + echo '[Mon May 6 19:51:41 CST 2024] <<<<<< START storage consumer in changefeed_error case >>>>>>' [Mon May 6 19:51:41 CST 2024] <<<<<< START storage consumer in changefeed_error case >>>>>> + cd /tmp/tidb_cdc_test/changefeed_error + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/changefeed_error + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/changefeed_error/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-32148?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' check_changefeed_state http://127.0.0.1:2379 changefeed-error failed [CDC:ErrStartTsBeforeGC] + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error + expected_state=failed + error_msg='[CDC:ErrStartTsBeforeGC]' + tls_dir='[CDC:ErrStartTsBeforeGC]' + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error -s + info='{ "upstream_id": 7365852910125063100, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 449575987528335361, "checkpoint_time": "2024-05-06 19:51:30.315", "error": { "time": "2024-05-06T19:51:39.772046337+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 449575987528335061 is earlier than or equal to GC safepoint at 449575987528335361" } }' + echo '{ "upstream_id": 7365852910125063100, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 449575987528335361, "checkpoint_time": "2024-05-06 19:51:30.315", "error": { "time": "2024-05-06T19:51:39.772046337+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 449575987528335061 is earlier than or equal to GC safepoint at 449575987528335361" } }' { "upstream_id": 7365852910125063100, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 449575987528335361, "checkpoint_time": "2024-05-06 19:51:30.315", "error": { "time": "2024-05-06T19:51:39.772046337+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 449575987528335061 is earlier than or equal to GC safepoint at 449575987528335361" } } ++ echo '{' '"upstream_id":' 7365852910125063100, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"failed",' '"checkpoint_tso":' 449575987528335361, '"checkpoint_time":' '"2024-05-06' '19:51:30.315",' '"error":' '{' '"time":' '"2024-05-06T19:51:39.772046337+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]fail' to create or maintain changefeed because start-ts 449575987528335061 is earlier than or equal to GC safepoint at '449575987528335361"' '}' '}' ++ jq -r .state + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ jq -r .error.message ++ echo '{' '"upstream_id":' 7365852910125063100, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"failed",' '"checkpoint_tso":' 449575987528335361, '"checkpoint_time":' '"2024-05-06' '19:51:30.315",' '"error":' '{' '"time":' '"2024-05-06T19:51:39.772046337+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]fail' to create or maintain changefeed because start-ts 449575987528335061 is earlier than or equal to GC safepoint at '449575987528335361"' '}' '}' + message='[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 449575987528335061 is earlier than or equal to GC safepoint at 449575987528335361' + [[ ! [CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 449575987528335061 is earlier than or equal to GC safepoint at 449575987528335361 =~ \[CDC:ErrStartTsBeforeGC] ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5594.out cli changefeed resume -c changefeed-error wait process cdc.test exit for 1-th time... [Mon May 6 19:51:41 CST 2024] <<<<<< START cdc server in changefeed_finish case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_finish.25482550.out server --log-file /tmp/tidb_cdc_test/changefeed_finish/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_finish/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fb2fa8000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-njdk7, pid:1347, start at 2024-05-06 19:51:40.784544586 +0800 CST m=+5.218373852 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:40.795 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:40.778 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:40.778 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fb2fa8000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-njdk7, pid:1347, start at 2024-05-06 19:51:40.784544586 +0800 CST m=+5.218373852 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:40.795 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:40.778 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:40.778 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fb32a40010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-njdk7, pid:1440, start at 2024-05-06 19:51:40.987507207 +0800 CST m=+5.337164908 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:40.994 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:40.969 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:40.969 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 2-th time... check diff failed 9-th time, retry later PASS coverage: 2.3% of statements in github.com/pingcap/tiflow/... Verifying downstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 86-th check, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table drop_tables.c not exists for 3-th check, retry later + set +x + tso='449575990328819713 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575990328819713 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:51:42 CST 2024] <<<<<< START cdc server in tiflash case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tiflash.26712673.out server --log-file /tmp/tidb_cdc_test/tiflash/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/tiflash/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 3-th time... [Pipeline] // cache VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fb44180015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-81vjm, pid:1277, start at 2024-05-06 19:51:42.133973655 +0800 CST m=+5.406331205 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:42.141 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:42.136 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:42.136 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:51:43 CST 2024] <<<<<< run test case split_region success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fb2a6c0010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w7kft, pid:1256, start at 2024-05-06 19:51:40.461424189 +0800 CST m=+5.485676438 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:40.469 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:40.443 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:40.443 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fb2a480015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w7kft, pid:1337, start at 2024-05-06 19:51:40.47917176 +0800 CST m=+5.441315816 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:40.486 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:40.485 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:40.485 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + set +x table changefeed_error.usertable not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_simple_handle_key_only_avro/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:51:43 CST 2024] <<<<<< run test case kafka_simple_handle_key_only_avro success! >>>>>> check diff successfully < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:51:43 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b633b83e-cb09-4a4e-912f-925fa340bb69 {"id":"b633b83e-cb09-4a4e-912f-925fa340bb69","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996301} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe8f5d34 b633b83e-cb09-4a4e-912f-925fa340bb69 /tidb/cdc/default/default/upstream/7365852967953359023 {"id":7365852967953359023,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b633b83e-cb09-4a4e-912f-925fa340bb69 {"id":"b633b83e-cb09-4a4e-912f-925fa340bb69","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996301} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe8f5d34 b633b83e-cb09-4a4e-912f-925fa340bb69 /tidb/cdc/default/default/upstream/7365852967953359023 {"id":7365852967953359023,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b633b83e-cb09-4a4e-912f-925fa340bb69 {"id":"b633b83e-cb09-4a4e-912f-925fa340bb69","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996301} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe8f5d34 b633b83e-cb09-4a4e-912f-925fa340bb69 /tidb/cdc/default/default/upstream/7365852967953359023 {"id":7365852967953359023,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.cli.2611.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-1485?protocol=canal-json&enable-tidb-extension=true' --server=127.0.0.1:8301 Create changefeed successfully! ID: 4e8e0192-0d5b-4b07-a20b-660f8782d3ff Info: {"upstream_id":7365852967953359023,"namespace":"default","id":"4e8e0192-0d5b-4b07-a20b-660f8782d3ff","sink_uri":"file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-1485?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:51:44.249872776+08:00","start_ts":449575991164010497,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575991164010497,"checkpoint_ts":449575991164010497,"checkpoint_time":"2024-05-06 19:51:44.184"} PASS ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 87-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table drop_tables.c not exists for 4-th check, retry later coverage: 2.7% of statements in github.com/pingcap/tiflow/... < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:51:44 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d3b30f26-3de2-492a-b595-c005616078ca {"id":"d3b30f26-3de2-492a-b595-c005616078ca","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996301} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe8ddd32 d3b30f26-3de2-492a-b595-c005616078ca /tidb/cdc/default/default/upstream/7365852967548068780 {"id":7365852967548068780,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d3b30f26-3de2-492a-b595-c005616078ca {"id":"d3b30f26-3de2-492a-b595-c005616078ca","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996301} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe8ddd32 d3b30f26-3de2-492a-b595-c005616078ca /tidb/cdc/default/default/upstream/7365852967548068780 {"id":7365852967548068780,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d3b30f26-3de2-492a-b595-c005616078ca {"id":"d3b30f26-3de2-492a-b595-c005616078ca","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996301} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe8ddd32 d3b30f26-3de2-492a-b595-c005616078ca /tidb/cdc/default/default/upstream/7365852967548068780 {"id":7365852967548068780,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Pipeline] } Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + workdir=/tmp/tidb_cdc_test/changefeed_finish + sink_uri='file:///tmp/tidb_cdc_test/changefeed_finish/storage_test/ticdc-changefeed-pause-resume-9705?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/changefeed_finish ++ date + echo '[Mon May 6 19:51:44 CST 2024] <<<<<< START storage consumer in changefeed_finish case >>>>>>' [Mon May 6 19:51:44 CST 2024] <<<<<< START storage consumer in changefeed_finish case >>>>>> + cd /tmp/tidb_cdc_test/changefeed_finish + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/changefeed_finish + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/changefeed_finish/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/changefeed_finish/storage_test/ticdc-changefeed-pause-resume-9705?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fb44180015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-81vjm, pid:1277, start at 2024-05-06 19:51:42.133973655 +0800 CST m=+5.406331205 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:42.141 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:42.136 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:42.136 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fb47280010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-81vjm, pid:1364, start at 2024-05-06 19:51:42.303102958 +0800 CST m=+5.519717402 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:42.310 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:42.282 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:42.282 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/owner_resign/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/owner_resign/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Mon May 6 19:51:44 CST 2024] <<<<<< START cdc server in batch_add_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.26882690.out server --log-file /tmp/tidb_cdc_test/batch_add_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/batch_add_table/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } check diff failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:51:45 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0f0983f1-bc25-44ab-aee8-2ea780696698 {"id":"0f0983f1-bc25-44ab-aee8-2ea780696698","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996302} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe8f3b3c 0f0983f1-bc25-44ab-aee8-2ea780696698 /tidb/cdc/default/default/upstream/7365852969171817338 {"id":7365852969171817338,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0f0983f1-bc25-44ab-aee8-2ea780696698 {"id":"0f0983f1-bc25-44ab-aee8-2ea780696698","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996302} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe8f3b3c 0f0983f1-bc25-44ab-aee8-2ea780696698 /tidb/cdc/default/default/upstream/7365852969171817338 {"id":7365852969171817338,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0f0983f1-bc25-44ab-aee8-2ea780696698 {"id":"0f0983f1-bc25-44ab-aee8-2ea780696698","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996302} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe8f3b3c 0f0983f1-bc25-44ab-aee8-2ea780696698 /tidb/cdc/default/default/upstream/7365852969171817338 {"id":7365852969171817338,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: 420db722-50b2-4d34-a367-13bcc1e950e0 Info: {"upstream_id":7365852969171817338,"namespace":"default","id":"420db722-50b2-4d34-a367-13bcc1e950e0","sink_uri":"file:///tmp/tidb_cdc_test/tiflash/storage_test/ticdc-tiflash-test-21286?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:51:45.760517707+08:00","start_ts":449575990328819713,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575990328819713,"checkpoint_ts":449575990328819713,"checkpoint_time":"2024-05-06 19:51:40.998"} + workdir=/tmp/tidb_cdc_test/tiflash + sink_uri='file:///tmp/tidb_cdc_test/tiflash/storage_test/ticdc-tiflash-test-21286?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/tiflash ++ date + echo '[Mon May 6 19:51:45 CST 2024] <<<<<< START storage consumer in tiflash case >>>>>>' [Mon May 6 19:51:45 CST 2024] <<<<<< START storage consumer in tiflash case >>>>>> + cd /tmp/tidb_cdc_test/tiflash + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/tiflash + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/tiflash/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/tiflash/storage_test/ticdc-tiflash-test-21286?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' + set +x + workdir=/tmp/tidb_cdc_test/processor_resolved_ts_fallback + sink_uri='file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-1485?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/processor_resolved_ts_fallback ++ date + echo '[Mon May 6 19:51:45 CST 2024] <<<<<< START storage consumer in processor_resolved_ts_fallback case >>>>>>' [Mon May 6 19:51:45 CST 2024] <<<<<< START storage consumer in processor_resolved_ts_fallback case >>>>>> + cd /tmp/tidb_cdc_test/processor_resolved_ts_fallback + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/processor_resolved_ts_fallback + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-1485?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' [Pipeline] // withEnv + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_session_done_during_task.cli.2725.out cli tso query --pd=http://127.0.0.1:2379 [Pipeline] } table changefeed_error.usertable not exists for 2-th check, retry later table cdc_tiflash_test.multi_data_type not exists for 1-th check, retry later [Pipeline] // node [Pipeline] } table ddl_manager.finish_mark not exists for 88-th check, retry later check diff failed 1-th time, retry later [Pipeline] // podTemplate ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table drop_tables.c not exists for 5-th check, retry later [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) 1 run task successfully table processor_resolved_ts_fallback.t1 not exists for 1-th check, retry later [Mon May 6 19:51:47 CST 2024] <<<<<< START cdc server in owner_resign case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.26202622.out server --log-file /tmp/tidb_cdc_test/owner_resign/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/owner_resign/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 2-th time, retry later + set +x + tso='449575991651336194 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575991651336194 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:51:47 CST 2024] <<<<<< START cdc server in capture_session_done_during_task case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorManagerHandleNewChangefeedDelay=sleep(2000)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_session_done_during_task.27692771.out server --log-file /tmp/tidb_cdc_test/capture_session_done_during_task/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/capture_session_done_during_task/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table changefeed_error.usertable not exists for 3-th check, retry later table ddl_manager.finish_mark not exists for 89-th check, retry later check diff failed 2-th time, retry later table drop_tables.c not exists for 6-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fb9944000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3, pid:1361, start at 2024-05-06 19:51:47.550512301 +0800 CST m=+5.417022647 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:47.557 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:47.537 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:47.537 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:51:47 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e9777339-d1f7-49d2-ad67-cd451a353e0c {"id":"e9777339-d1f7-49d2-ad67-cd451a353e0c","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996305} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe9fc069 e9777339-d1f7-49d2-ad67-cd451a353e0c /tidb/cdc/default/default/upstream/7365852990836052600 {"id":7365852990836052600,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e9777339-d1f7-49d2-ad67-cd451a353e0c {"id":"e9777339-d1f7-49d2-ad67-cd451a353e0c","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996305} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe9fc069 e9777339-d1f7-49d2-ad67-cd451a353e0c /tidb/cdc/default/default/upstream/7365852990836052600 {"id":7365852990836052600,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e9777339-d1f7-49d2-ad67-cd451a353e0c {"id":"e9777339-d1f7-49d2-ad67-cd451a353e0c","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996305} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe9fc069 e9777339-d1f7-49d2-ad67-cd451a353e0c /tidb/cdc/default/default/upstream/7365852990836052600 {"id":7365852990836052600,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.cli.2744.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/batch_add_table/storage_test/ticdc-batch-add-table-test-6557?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: 84b36aa0-cab3-4253-8913-f5679e5711ea Info: {"upstream_id":7365852990836052600,"namespace":"default","id":"84b36aa0-cab3-4253-8913-f5679e5711ea","sink_uri":"file:///tmp/tidb_cdc_test/batch_add_table/storage_test/ticdc-batch-add-table-test-6557?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:51:48.363698334+08:00","start_ts":449575992237228034,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575992237228034,"checkpoint_ts":449575992237228034,"checkpoint_time":"2024-05-06 19:51:48.278"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table cdc_tiflash_test.multi_data_type not exists for 2-th check, retry later table processor_resolved_ts_fallback.t1 not exists for 2-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/batch_add_table + sink_uri='file:///tmp/tidb_cdc_test/batch_add_table/storage_test/ticdc-batch-add-table-test-6557?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/batch_add_table ++ date + echo '[Mon May 6 19:51:49 CST 2024] <<<<<< START storage consumer in batch_add_table case >>>>>>' [Mon May 6 19:51:49 CST 2024] <<<<<< START storage consumer in batch_add_table case >>>>>> + cd /tmp/tidb_cdc_test/batch_add_table + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/batch_add_table + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/batch_add_table/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/batch_add_table/storage_test/ticdc-batch-add-table-test-6557?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > table batch_add_table.finish_mark not exists for 1-th check, retry later table changefeed_error.usertable not exists for 4-th check, retry later table drop_tables.c not exists for 7-th check, retry later check diff failed 3-th time, retry later < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:51:50 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a8c96bb4-88b2-4a96-a776-a9800b328e81 {"id":"a8c96bb4-88b2-4a96-a776-a9800b328e81","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996307} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbea63a32 a8c96bb4-88b2-4a96-a776-a9800b328e81 /tidb/cdc/default/default/upstream/7365852992115240155 {"id":7365852992115240155,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a8c96bb4-88b2-4a96-a776-a9800b328e81 {"id":"a8c96bb4-88b2-4a96-a776-a9800b328e81","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996307} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbea63a32 a8c96bb4-88b2-4a96-a776-a9800b328e81 /tidb/cdc/default/default/upstream/7365852992115240155 {"id":7365852992115240155,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a8c96bb4-88b2-4a96-a776-a9800b328e81 {"id":"a8c96bb4-88b2-4a96-a776-a9800b328e81","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996307} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbea63a32 a8c96bb4-88b2-4a96-a776-a9800b328e81 /tidb/cdc/default/default/upstream/7365852992115240155 {"id":7365852992115240155,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc cli capture list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' 1 run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.cli.2722.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-8883?protocol=canal-json&enable-tidb-extension=true' --server=127.0.0.1:8301 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table cdc_tiflash_test.multi_data_type not exists for 3-th check, retry later table ddl_manager.finish_mark not exists for 90-th check, retry later check diff failed 3-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fb9944000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3, pid:1361, start at 2024-05-06 19:51:47.550512301 +0800 CST m=+5.417022647 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:47.557 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:47.537 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:47.537 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fb9a000013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3, pid:1437, start at 2024-05-06 19:51:47.609211861 +0800 CST m=+5.420532005 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:47.616 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:47.584 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:47.584 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/simple/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/simple/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/simple/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/simple/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/simple/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Create changefeed successfully! ID: d76b659f-2a16-4cbe-89f9-8173fe4be567 Info: {"upstream_id":7365852992115240155,"namespace":"default","id":"d76b659f-2a16-4cbe-89f9-8173fe4be567","sink_uri":"file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-8883?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:51:50.739873053+08:00","start_ts":449575992855363586,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575992855363586,"checkpoint_ts":449575992855363586,"checkpoint_time":"2024-05-06 19:51:50.636"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:51:50 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e797b5e9-6f33-484c-8049-2fa26be4372c {"id":"e797b5e9-6f33-484c-8049-2fa26be4372c","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996307} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe9d7753 e797b5e9-6f33-484c-8049-2fa26be4372c /tidb/cdc/default/default/upstream/7365852983797276337 {"id":7365852983797276337,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e797b5e9-6f33-484c-8049-2fa26be4372c {"id":"e797b5e9-6f33-484c-8049-2fa26be4372c","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996307} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe9d7753 e797b5e9-6f33-484c-8049-2fa26be4372c /tidb/cdc/default/default/upstream/7365852983797276337 {"id":7365852983797276337,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e797b5e9-6f33-484c-8049-2fa26be4372c {"id":"e797b5e9-6f33-484c-8049-2fa26be4372c","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996307} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe9d7753 e797b5e9-6f33-484c-8049-2fa26be4372c /tidb/cdc/default/default/upstream/7365852983797276337 {"id":7365852983797276337,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table processor_resolved_ts_fallback.t1 not exists for 3-th check, retry later table changefeed_error.usertable exists check diff successfully ***************** properties ***************** "mysql.db"="changefeed_error" "requestdistribution"="uniform" "mysql.port"="4000" "readallfields"="true" "insertproportion"="0" "readproportion"="0" "mysql.host"="127.0.0.1" "recordcount"="20" "workload"="core" "operationcount"="0" "threadcount"="4" "dotransactions"="false" "scanproportion"="0" "updateproportion"="0" "mysql.user"="root" ********************************************** Run finished, takes 5.590479ms INSERT - Takes(s): 0.0, Count: 20, OPS: 5115.6, Avg(us): 961, Min(us): 456, Max(us): 2634, 95th(us): 3000, 99th(us): 3000 check diff successfully {"id":"c4c334b5-9074-4a15-9a6e-abf2a6371f87","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996296} check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/c4c334b5-9074-4a15-9a6e-abf2a6371f87 + [[ /tidb/cdc/default/__cdc_meta__/capture/c4c334b5-9074-4a15-9a6e-abf2a6371f87 =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/c4c334b5-9074-4a15-9a6e-abf2a6371f87' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/c4c334b5-9074-4a15-9a6e-abf2a6371f87 + echo 'check failed' check failed + exit 1 run task failed 1-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_simple_claim_check/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:51:51 CST 2024] <<<<<< run test case kafka_simple_claim_check success! >>>>>> table batch_add_table.finish_mark not exists for 2-th check, retry later + workdir=/tmp/tidb_cdc_test/capture_session_done_during_task + sink_uri='file:///tmp/tidb_cdc_test/capture_session_done_during_task/storage_test/ticdc-capture-session-done-during-task-16102?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/capture_session_done_during_task ++ date + echo '[Mon May 6 19:51:51 CST 2024] <<<<<< START storage consumer in capture_session_done_during_task case >>>>>>' [Mon May 6 19:51:51 CST 2024] <<<<<< START storage consumer in capture_session_done_during_task case >>>>>> + cd /tmp/tidb_cdc_test/capture_session_done_during_task + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/capture_session_done_during_task + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/capture_session_done_during_task/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/capture_session_done_during_task/storage_test/ticdc-capture-session-done-during-task-16102?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' lease 22318f4dbe9d7753 revoked check diff failed 4-th time, retry later + set +x + workdir=/tmp/tidb_cdc_test/owner_resign + sink_uri='file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-8883?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/owner_resign ++ date + echo '[Mon May 6 19:51:52 CST 2024] <<<<<< START storage consumer in owner_resign case >>>>>>' [Mon May 6 19:51:52 CST 2024] <<<<<< START storage consumer in owner_resign case >>>>>> + cd /tmp/tidb_cdc_test/owner_resign + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/owner_resign + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/owner_resign/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-8883?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table cdc_tiflash_test.multi_data_type not exists for 4-th check, retry later cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' table ddl_manager.finish_mark not exists for 91-th check, retry later check diff failed 4-th time, retry later table drop_tables.c not exists for 8-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.cli.2694.out cli tso query --pd=http://127.0.0.1:2379 1 run task successfully table owner_resign.t1 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fbda30000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l, pid:7232, start at 2024-05-06 19:51:51.701368439 +0800 CST m=+5.383303337 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:51.711 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:51.692 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:51.692 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table capture_session_done_during_task.t exists check diff failed 1-th time, retry later table processor_resolved_ts_fallback.t1 not exists for 4-th check, retry later check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/c4c334b5-9074-4a15-9a6e-abf2a6371f87 + [[ /tidb/cdc/default/__cdc_meta__/capture/c4c334b5-9074-4a15-9a6e-abf2a6371f87 =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/c4c334b5-9074-4a15-9a6e-abf2a6371f87' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/c4c334b5-9074-4a15-9a6e-abf2a6371f87 + echo 'check failed' check failed + exit 1 run task failed 2-th time, retry later table cdc_tiflash_test.multi_data_type not exists for 5-th check, retry later table ddl_manager.finish_mark not exists for 92-th check, retry later check diff failed 5-th time, retry later table batch_add_table.finish_mark not exists for 3-th check, retry later table drop_tables.c not exists for 9-th check, retry later + set +x + tso='449575993419497473 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575993419497473 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:51:54 CST 2024] <<<<<< START cdc server in simple case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.27472749.out server --log-file /tmp/tidb_cdc_test/simple/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/simple/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 5-th time, retry later table owner_resign.t1 not exists for 2-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_basic/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fbfb50000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-pst3l, pid:1263, start at 2024-05-06 19:51:53.851939668 +0800 CST m=+15.631252549 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:53.919 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:53.961 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:53.961 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fbfb50000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-pst3l, pid:1263, start at 2024-05-06 19:51:53.851939668 +0800 CST m=+15.631252549 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:53.919 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:53.961 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:53.961 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fbf33c000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-pst3l, pid:1307, start at 2024-05-06 19:51:53.337244347 +0800 CST m=+14.990328663 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:53.380 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:53.396 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:53.396 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... table processor_resolved_ts_fallback.t1 not exists for 5-th check, retry later Logging trace to /tmp/tidb_cdc_test/resolve_lock/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/resolve_lock/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } \033[0;36m<<< Run all test success >>>\033[0m VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fbda30000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l, pid:7232, start at 2024-05-06 19:51:51.701368439 +0800 CST m=+5.383303337 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:51.711 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:51.692 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:51.692 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fbdae00015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l, pid:7316, start at 2024-05-06 19:51:51.779117269 +0800 CST m=+5.409426199 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:51.786 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:51.787 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:51.787 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_capture/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_capture/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc already exists) check diff failed 2-th time, retry later [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_simple_claim_check_avro/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:51:55 CST 2024] <<<<<< run test case kafka_simple_claim_check_avro success! >>>>>> [Pipeline] // stage [Pipeline] } [Pipeline] // container table cdc_tiflash_test.multi_data_type exists [Pipeline] } table ddl_manager.finish_mark exists check diff successfully [Pipeline] // withEnv [Pipeline] } table batch_add_table.finish_mark not exists for 4-th check, retry later table drop_tables.c not exists for 10-th check, retry later check diff successfully [Pipeline] // node check diff failed 6-th time, retry later [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } wait process cdc.test exit for 1-th time... [Pipeline] // stage [Pipeline] } check diff successfully wait process cdc.test exit for 1-th time... table processor_resolved_ts_fallback.t1 exists [Mon May 6 19:51:56 CST 2024] <<<<<< START cdc server in processor_resolved_ts_fallback case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.27732775.out server --log-file /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table owner_resign.t1 not exists for 3-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.cli.2552.out cli tso query --pd=http://127.0.0.1:2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.cli.8584.out cli tso query --pd=http://127.0.0.1:2379 wait process cdc.test exit for 2-th time... check diff failed 3-th time, retry later wait process cdc.test exit for 2-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:51:57 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ffaf6d2b-0ac7-4b28-aa99-034f390f1951 {"id":"ffaf6d2b-0ac7-4b28-aa99-034f390f1951","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996314} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbeb9604e ffaf6d2b-0ac7-4b28-aa99-034f390f1951 /tidb/cdc/default/default/upstream/7365853018316735784 {"id":7365853018316735784,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ffaf6d2b-0ac7-4b28-aa99-034f390f1951 {"id":"ffaf6d2b-0ac7-4b28-aa99-034f390f1951","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996314} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbeb9604e ffaf6d2b-0ac7-4b28-aa99-034f390f1951 /tidb/cdc/default/default/upstream/7365853018316735784 {"id":7365853018316735784,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ffaf6d2b-0ac7-4b28-aa99-034f390f1951 {"id":"ffaf6d2b-0ac7-4b28-aa99-034f390f1951","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996314} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbeb9604e ffaf6d2b-0ac7-4b28-aa99-034f390f1951 /tidb/cdc/default/default/upstream/7365853018316735784 {"id":7365853018316735784,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.cli.2806.out cli changefeed create --start-ts=449575993419497473 '--sink-uri=file:///tmp/tidb_cdc_test/simple/storage_test/ticdc-simple-test-13500?protocol=canal-json&enable-tidb-extension=true' wait process cdc.test exit for 3-th time... check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/c4c334b5-9074-4a15-9a6e-abf2a6371f87 + [[ /tidb/cdc/default/__cdc_meta__/capture/c4c334b5-9074-4a15-9a6e-abf2a6371f87 =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/c4c334b5-9074-4a15-9a6e-abf2a6371f87' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/c4c334b5-9074-4a15-9a6e-abf2a6371f87 + echo 'check failed' check failed + exit 1 run task failed 3-th time, retry later wait process cdc.test exit for 3-th time... table batch_add_table.finish_mark not exists for 5-th check, retry later table drop_tables.c exists check diff successfully Create changefeed successfully! ID: e6d76bf0-6cf4-4da5-a729-bab47710b931 Info: {"upstream_id":7365853018316735784,"namespace":"default","id":"e6d76bf0-6cf4-4da5-a729-bab47710b931","sink_uri":"file:///tmp/tidb_cdc_test/simple/storage_test/ticdc-simple-test-13500?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:51:58.061426084+08:00","start_ts":449575993419497473,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575993419497473,"checkpoint_ts":449575993419497473,"checkpoint_time":"2024-05-06 19:51:52.788"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... check diff failed 7-th time, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:51:58 CST 2024] <<<<<< run test case tiflash success! >>>>>> wait process cdc.test exit for 4-th time... cdc.test: no process found wait process cdc.test exit for 5-th time... process cdc.test already exit [Mon May 6 19:51:58 CST 2024] <<<<<< run test case ddl_manager success! >>>>>> wait process cdc.test exit for 1-th time... + set +x + tso='449575994678837249 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575994678837249 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ***************** properties ***************** "threadcount"="2" "requestdistribution"="uniform" "insertproportion"="0" "recordcount"="10" "readproportion"="0" "dotransactions"="false" "readallfields"="true" "mysql.port"="4000" "updateproportion"="0" "mysql.user"="root" "operationcount"="0" "mysql.host"="127.0.0.1" "mysql.db"="multi_capture_1" "scanproportion"="0" "workload"="core" ********************************************** =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_adapter_compatibility/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:51:58 CST 2024] <<<<<< run test case canal_json_adapter_compatibility success! >>>>>> check diff failed 4-th time, retry later wait process cdc.test exit for 2-th time... Run finished, takes 8.950484ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1963.9, Avg(us): 1705, Min(us): 1010, Max(us): 3790, 95th(us): 4000, 99th(us): 4000 ***************** properties ***************** "readallfields"="true" "insertproportion"="0" "mysql.port"="4000" "updateproportion"="0" "operationcount"="0" "requestdistribution"="uniform" "recordcount"="10" "scanproportion"="0" "readproportion"="0" "threadcount"="2" "workload"="core" "dotransactions"="false" "mysql.user"="root" "mysql.host"="127.0.0.1" "mysql.db"="multi_capture_2" ********************************************** Run finished, takes 9.81721ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1874.9, Avg(us): 1873, Min(us): 1144, Max(us): 4382, 95th(us): 5000, 99th(us): 5000 table owner_resign.t1 not exists for 4-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/simple + sink_uri='file:///tmp/tidb_cdc_test/simple/storage_test/ticdc-simple-test-13500?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/simple ++ date + echo '[Mon May 6 19:51:59 CST 2024] <<<<<< START storage consumer in simple case >>>>>>' [Mon May 6 19:51:59 CST 2024] <<<<<< START storage consumer in simple case >>>>>> + cd /tmp/tidb_cdc_test/simple + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/simple + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/simple/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/simple/storage_test/ticdc-simple-test-13500?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' succeed to verify meta placement rules ***************** properties ***************** "threadcount"="2" "dotransactions"="false" "mysql.host"="127.0.0.1" "scanproportion"="0" "operationcount"="0" "recordcount"="10" "readallfields"="true" "mysql.user"="root" "insertproportion"="0" "readproportion"="0" "requestdistribution"="uniform" "workload"="core" "mysql.port"="4000" "updateproportion"="0" "mysql.db"="multi_capture_3" ********************************************** Run finished, takes 10.665603ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1585.3, Avg(us): 2050, Min(us): 1156, Max(us): 4251, 95th(us): 5000, 99th(us): 5000 + set +x + tso='449575994737295362 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + awk -F ' ' '{print $1}' + echo 449575994737295362 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + set +x [Mon May 6 19:51:59 CST 2024] <<<<<< START cdc server in resolve_lock case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.25742576.out server --log-file /tmp/tidb_cdc_test/resolve_lock/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/resolve_lock/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo '' + '[' 0 -eq 50 ']' + sleep 3 ERROR 1146 (42S02) at line 1: Table 'test.simple1' doesn't exist check data failed 1-th time, retry later wait process cdc.test exit for 3-th time... ***************** properties ***************** "updateproportion"="0" "mysql.port"="4000" "dotransactions"="false" "readallfields"="true" "scanproportion"="0" "mysql.user"="root" "workload"="core" "mysql.host"="127.0.0.1" "requestdistribution"="uniform" "threadcount"="2" "mysql.db"="multi_capture_4" "readproportion"="0" "insertproportion"="0" "operationcount"="0" "recordcount"="10" ********************************************** Run finished, takes 9.521284ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1856.4, Avg(us): 1814, Min(us): 979, Max(us): 4029, 95th(us): 5000, 99th(us): 5000 [Mon May 6 19:51:59 CST 2024] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.87118713.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table batch_add_table.finish_mark not exists for 6-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/csv_storage_basic Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... check diff failed 8-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:52:00 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/4e8e0192-0d5b-4b07-a20b-660f8782d3ff {UpstreamID:7365852967953359023 Namespace:default ID:4e8e0192-0d5b-4b07-a20b-660f8782d3ff SinkURI:file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-1485?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:51:44.249872776 +0800 CST StartTs:449575991164010497 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003a15680 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575991177117700} {CheckpointTs:449575995161968648 MinTableBarrierTs:449575995161968648 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b633b83e-cb09-4a4e-912f-925fa340bb69 {"id":"b633b83e-cb09-4a4e-912f-925fa340bb69","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996301} /tidb/cdc/default/__cdc_meta__/capture/bc767331-f7c7-41e3-8c20-897a40f7776c {"id":"bc767331-f7c7-41e3-8c20-897a40f7776c","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996317} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe8f5d34 b633b83e-cb09-4a4e-912f-925fa340bb69 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe8f5df6 bc767331-f7c7-41e3-8c20-897a40f7776c /tidb/cdc/default/default/changefeed/info/4e8e0192-0d5b-4b07-a20b-660f8782d3ff {"upstream-id":7365852967953359023,"namespace":"default","changefeed-id":"4e8e0192-0d5b-4b07-a20b-660f8782d3ff","sink-uri":"file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-1485?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:51:44.249872776+08:00","start-ts":449575991164010497,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575991177117700} /tidb/cdc/default/default/changefeed/status/4e8e0192-0d5b-4b07-a20b-660f8782d3ff {"checkpoint-ts":449575995161968648,"min-table-barrier-ts":449575995161968648,"admin-job-type":0} /tidb/cdc/default/default/task/position/b633b83e-cb09-4a4e-912f-925fa340bb69/4e8e0192-0d5b-4b07-a20b-660f8782d3ff {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/bc767331-f7c7-41e3-8c20-897a40f7776c/4e8e0192-0d5b-4b07-a20b-660f8782d3ff {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852967953359023 {"id":7365852967953359023,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/4e8e0192-0d5b-4b07-a20b-660f8782d3ff {UpstreamID:7365852967953359023 Namespace:default ID:4e8e0192-0d5b-4b07-a20b-660f8782d3ff SinkURI:file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-1485?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:51:44.249872776 +0800 CST StartTs:449575991164010497 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003a15680 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575991177117700} {CheckpointTs:449575995161968648 MinTableBarrierTs:449575995161968648 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b633b83e-cb09-4a4e-912f-925fa340bb69 {"id":"b633b83e-cb09-4a4e-912f-925fa340bb69","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996301} /tidb/cdc/default/__cdc_meta__/capture/bc767331-f7c7-41e3-8c20-897a40f7776c {"id":"bc767331-f7c7-41e3-8c20-897a40f7776c","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996317} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe8f5d34 b633b83e-cb09-4a4e-912f-925fa340bb69 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe8f5df6 bc767331-f7c7-41e3-8c20-897a40f7776c /tidb/cdc/default/default/changefeed/info/4e8e0192-0d5b-4b07-a20b-660f8782d3ff {"upstream-id":7365852967953359023,"namespace":"default","changefeed-id":"4e8e0192-0d5b-4b07-a20b-660f8782d3ff","sink-uri":"file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-1485?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:51:44.249872776+08:00","start-ts":449575991164010497,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575991177117700} /tidb/cdc/default/default/changefeed/status/4e8e0192-0d5b-4b07-a20b-660f8782d3ff {"checkpoint-ts":449575995161968648,"min-table-barrier-ts":449575995161968648,"admin-job-type":0} /tidb/cdc/default/default/task/position/b633b83e-cb09-4a4e-912f-925fa340bb69/4e8e0192-0d5b-4b07-a20b-660f8782d3ff {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/bc767331-f7c7-41e3-8c20-897a40f7776c/4e8e0192-0d5b-4b07-a20b-660f8782d3ff {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852967953359023 {"id":7365852967953359023,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/4e8e0192-0d5b-4b07-a20b-660f8782d3ff {UpstreamID:7365852967953359023 Namespace:default ID:4e8e0192-0d5b-4b07-a20b-660f8782d3ff SinkURI:file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-1485?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:51:44.249872776 +0800 CST StartTs:449575991164010497 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003a15680 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575991177117700} {CheckpointTs:449575995161968648 MinTableBarrierTs:449575995161968648 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b633b83e-cb09-4a4e-912f-925fa340bb69 {"id":"b633b83e-cb09-4a4e-912f-925fa340bb69","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996301} /tidb/cdc/default/__cdc_meta__/capture/bc767331-f7c7-41e3-8c20-897a40f7776c {"id":"bc767331-f7c7-41e3-8c20-897a40f7776c","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996317} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe8f5d34 b633b83e-cb09-4a4e-912f-925fa340bb69 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe8f5df6 bc767331-f7c7-41e3-8c20-897a40f7776c /tidb/cdc/default/default/changefeed/info/4e8e0192-0d5b-4b07-a20b-660f8782d3ff {"upstream-id":7365852967953359023,"namespace":"default","changefeed-id":"4e8e0192-0d5b-4b07-a20b-660f8782d3ff","sink-uri":"file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-1485?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:51:44.249872776+08:00","start-ts":449575991164010497,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575991177117700} /tidb/cdc/default/default/changefeed/status/4e8e0192-0d5b-4b07-a20b-660f8782d3ff {"checkpoint-ts":449575995161968648,"min-table-barrier-ts":449575995161968648,"admin-job-type":0} /tidb/cdc/default/default/task/position/b633b83e-cb09-4a4e-912f-925fa340bb69/4e8e0192-0d5b-4b07-a20b-660f8782d3ff {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/bc767331-f7c7-41e3-8c20-897a40f7776c/4e8e0192-0d5b-4b07-a20b-660f8782d3ff {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852967953359023 {"id":7365852967953359023,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:52:00 CST 2024] <<<<<< run test case drop_many_tables success! >>>>>> table processor_resolved_ts_fallback.t2 not exists for 1-th check, retry later table owner_resign.t1 not exists for 5-th check, retry later ERROR 1146 (42S02) at line 1: Table 'test.simple1' doesn't exist check data failed 2-th time, retry later check diff failed 5-th time, retry later table batch_add_table.finish_mark exists check diff successfully check diff failed 9-th time, retry later wait process cdc.test exit for 1-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:52:02 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0f6c3b13-8e0a-41b4-aa7d-30b74b132330 {"id":"0f6c3b13-8e0a-41b4-aa7d-30b74b132330","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996320} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbeab6658 0f6c3b13-8e0a-41b4-aa7d-30b74b132330 /tidb/cdc/default/default/upstream/7365852987147108412 {"id":7365852987147108412,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0f6c3b13-8e0a-41b4-aa7d-30b74b132330 {"id":"0f6c3b13-8e0a-41b4-aa7d-30b74b132330","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996320} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbeab6658 0f6c3b13-8e0a-41b4-aa7d-30b74b132330 /tidb/cdc/default/default/upstream/7365852987147108412 {"id":7365852987147108412,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0f6c3b13-8e0a-41b4-aa7d-30b74b132330 {"id":"0f6c3b13-8e0a-41b4-aa7d-30b74b132330","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996320} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbeab6658 0f6c3b13-8e0a-41b4-aa7d-30b74b132330 /tidb/cdc/default/default/upstream/7365852987147108412 {"id":7365852987147108412,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.cli.2608.out cli changefeed create --start-ts=449575994737295362 '--sink-uri=file:///tmp/tidb_cdc_test/resolve_lock/storage_test/ticdc-resolve-lock-test-26657?protocol=canal-json&enable-tidb-extension=true' table processor_resolved_ts_fallback.t2 not exists for 2-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:52:02 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66b56055-3a26-4577-bfbe-45c9145c65d6 {"id":"66b56055-3a26-4577-bfbe-45c9145c65d6","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996320} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbecb8f8f 66b56055-3a26-4577-bfbe-45c9145c65d6 /tidb/cdc/default/default/upstream/7365853026311312504 {"id":7365853026311312504,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66b56055-3a26-4577-bfbe-45c9145c65d6 {"id":"66b56055-3a26-4577-bfbe-45c9145c65d6","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996320} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbecb8f8f 66b56055-3a26-4577-bfbe-45c9145c65d6 /tidb/cdc/default/default/upstream/7365853026311312504 {"id":7365853026311312504,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66b56055-3a26-4577-bfbe-45c9145c65d6 {"id":"66b56055-3a26-4577-bfbe-45c9145c65d6","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996320} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbecb8f8f 66b56055-3a26-4577-bfbe-45c9145c65d6 /tidb/cdc/default/default/upstream/7365853026311312504 {"id":7365853026311312504,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Mon May 6 19:52:02 CST 2024] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.87778779.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 2-th time... table owner_resign.t1 exists Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release wait process cdc.test exit for 3-th time... check diff failed 1-th time, retry later check diff successfully check diff failed 1-th time, retry later Create changefeed successfully! ID: 7527f263-e0aa-43fb-8692-dc260d055074 Info: {"upstream_id":7365852987147108412,"namespace":"default","id":"7527f263-e0aa-43fb-8692-dc260d055074","sink_uri":"file:///tmp/tidb_cdc_test/resolve_lock/storage_test/ticdc-resolve-lock-test-26657?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:52:03.744656891+08:00","start_ts":449575994737295362,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575994737295362,"checkpoint_ts":449575994737295362,"checkpoint_time":"2024-05-06 19:51:57.815"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:52:04 CST 2024] <<<<<< run test case batch_add_table success! >>>>>> ERROR 1146 (42S02) at line 1: Table 'test.simple1' doesn't exist check data failed 3-th time, retry later check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info= + [[ '' =~ capture ]] + echo 'check pass' check pass + exit 0 run task successfully check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Mon May 6 19:52:03 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/NewChangefeedRetryError=return(true)' + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.58475849.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff successfully Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later table processor_resolved_ts_fallback.t2 not exists for 3-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/resolve_lock + sink_uri='file:///tmp/tidb_cdc_test/resolve_lock/storage_test/ticdc-resolve-lock-test-26657?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/resolve_lock ++ date + echo '[Mon May 6 19:52:05 CST 2024] <<<<<< START storage consumer in resolve_lock case >>>>>>' [Mon May 6 19:52:05 CST 2024] <<<<<< START storage consumer in resolve_lock case >>>>>> + cd /tmp/tidb_cdc_test/resolve_lock + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/resolve_lock + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/resolve_lock/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/resolve_lock/storage_test/ticdc-resolve-lock-test-26657?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20240409022718-714958ccd4d5 go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20240415145106-cd9c676e9ba4 go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20240410110152-5fc42c9be2f5 go: downloading github.com/pingcap/kvproto v0.0.0-20240227073058-929ab83f9754 go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading github.com/tikv/pd/client v0.0.0-20240322051414-fb9e2d561b6e go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/pingcap/tidb-tools v0.0.0-20240305021104-9f9bea84490b go: downloading github.com/BurntSushi/toml v1.3.2 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading go.uber.org/atomic v1.11.0 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/pingcap/failpoint v0.0.0-20220801062533-2eaa32854a6c go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading google.golang.org/grpc v1.62.1 check diff failed 2-th time, retry later ERROR 1146 (42S02) at line 1: Table 'test.simple1' doesn't exist check data failed 4-th time, retry later go: downloading github.com/google/uuid v1.6.0 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/golang/protobuf v1.5.4 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/google/btree v1.1.2 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/prometheus/client_golang v1.19.0 go: downloading golang.org/x/sync v0.7.0 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 go: downloading github.com/docker/go-units v0.5.0 go: downloading google.golang.org/protobuf v1.33.0 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/prometheus/common v0.52.2 go: downloading github.com/prometheus/procfs v0.13.0 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:52:06 GMT < Content-Length: 1299 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66b56055-3a26-4577-bfbe-45c9145c65d6 {"id":"66b56055-3a26-4577-bfbe-45c9145c65d6","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996320} /tidb/cdc/default/__cdc_meta__/capture/f8c92dab-05fc-4d43-a50f-8f4048ae0ddb {"id":"f8c92dab-05fc-4d43-a50f-8f4048ae0ddb","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996323} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbecb8f8f 66b56055-3a26-4577-bfbe-45c9145c65d6 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbecb8fc4 f8c92dab-05fc-4d43-a50f-8f4048ae0ddb /tidb/cdc/default/default/upstream/7365853026311312504 {"id":7365853026311312504,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66b56055-3a26-4577-bfbe-45c9145c65d6 {"id":"66b56055-3a26-4577-bfbe-45c9145c65d6","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996320} /tidb/cdc/default/__cdc_meta__/capture/f8c92dab-05fc-4d43-a50f-8f4048ae0ddb {"id":"f8c92dab-05fc-4d43-a50f-8f4048ae0ddb","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996323} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbecb8f8f 66b56055-3a26-4577-bfbe-45c9145c65d6 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbecb8fc4 f8c92dab-05fc-4d43-a50f-8f4048ae0ddb /tidb/cdc/default/default/upstream/7365853026311312504 {"id":7365853026311312504,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66b56055-3a26-4577-bfbe-45c9145c65d6 {"id":"66b56055-3a26-4577-bfbe-45c9145c65d6","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996320} /tidb/cdc/default/__cdc_meta__/capture/f8c92dab-05fc-4d43-a50f-8f4048ae0ddb {"id":"f8c92dab-05fc-4d43-a50f-8f4048ae0ddb","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996323} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbecb8f8f 66b56055-3a26-4577-bfbe-45c9145c65d6 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbecb8fc4 f8c92dab-05fc-4d43-a50f-8f4048ae0ddb /tidb/cdc/default/default/upstream/7365853026311312504 {"id":7365853026311312504,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Mon May 6 19:52:06 CST 2024] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.88308832.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc3.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data3 --cluster-id default --addr 127.0.0.1:8303 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8303 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8303; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading golang.org/x/sys v0.19.0 go: downloading golang.org/x/exp v0.0.0-20240409090435-93d18d7e34b8 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240401170217-c3f982113cda go: downloading golang.org/x/net v0.24.0 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:52:06 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/21a1512a-6a96-4714-b7b8-0ef292553405 {"id":"21a1512a-6a96-4714-b7b8-0ef292553405","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996326} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe616582 21a1512a-6a96-4714-b7b8-0ef292553405 /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7365852910125063100,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-32148?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:51:39.698726487+08:00","start-ts":449575987528335361,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"pending","error":{"time":"2024-05-06T19:52:06.764038956+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575997070114825} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":449575992876072964,"min-table-barrier-ts":449575992876072964,"admin-job-type":1} /tidb/cdc/default/default/upstream/7365852910125063100 {"id":7365852910125063100,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/21a1512a-6a96-4714-b7b8-0ef292553405 {"id":"21a1512a-6a96-4714-b7b8-0ef292553405","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996326} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe616582 21a1512a-6a96-4714-b7b8-0ef292553405 /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7365852910125063100,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-32148?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:51:39.698726487+08:00","start-ts":449575987528335361,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"pending","error":{"time":"2024-05-06T19:52:06.764038956+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575997070114825} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":449575992876072964,"min-table-barrier-ts":449575992876072964,"admin-job-type":1} /tidb/cdc/default/default/upstream/7365852910125063100 {"id":7365852910125063100,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/21a1512a-6a96-4714-b7b8-0ef292553405 {"id":"21a1512a-6a96-4714-b7b8-0ef292553405","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996326} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe616582 21a1512a-6a96-4714-b7b8-0ef292553405 /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7365852910125063100,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-32148?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:51:39.698726487+08:00","start-ts":449575987528335361,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"pending","error":{"time":"2024-05-06T19:52:06.764038956+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575997070114825} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":449575992876072964,"min-table-barrier-ts":449575992876072964,"admin-job-type":1} /tidb/cdc/default/default/upstream/7365852910125063100 {"id":7365852910125063100,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-error warning failpoint injected retriable error + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error + expected_state=warning + error_msg=failpoint + tls_dir=error + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error -s check diff failed 1-th time, retry later + info='{ "upstream_id": 7365852910125063100, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 449575992876072964, "checkpoint_time": "2024-05-06 19:51:50.715", "error": { "time": "2024-05-06T19:52:06.764038956+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } }' + echo '{ "upstream_id": 7365852910125063100, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 449575992876072964, "checkpoint_time": "2024-05-06 19:51:50.715", "error": { "time": "2024-05-06T19:52:06.764038956+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } }' { "upstream_id": 7365852910125063100, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 449575992876072964, "checkpoint_time": "2024-05-06 19:51:50.715", "error": { "time": "2024-05-06T19:52:06.764038956+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } } ++ echo '{' '"upstream_id":' 7365852910125063100, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"warning",' '"checkpoint_tso":' 449575992876072964, '"checkpoint_time":' '"2024-05-06' '19:51:50.715",' '"error":' '{' '"time":' '"2024-05-06T19:52:06.764038956+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrOwnerUnknown",' '"message":' '"failpoint' injected retriable 'error"' '}' '}' ++ jq -r .state + state=warning + [[ ! warning == \w\a\r\n\i\n\g ]] ++ echo '{' '"upstream_id":' 7365852910125063100, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"warning",' '"checkpoint_tso":' 449575992876072964, '"checkpoint_time":' '"2024-05-06' '19:51:50.715",' '"error":' '{' '"time":' '"2024-05-06T19:52:06.764038956+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrOwnerUnknown",' '"message":' '"failpoint' injected retriable 'error"' '}' '}' ++ jq -r .error.message + message='failpoint injected retriable error' + [[ ! failpoint injected retriable error =~ failpoint ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5946.out cli changefeed remove -c changefeed-error go: downloading golang.org/x/text v0.14.0 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_resolved_ts_fallback.t2 exists table processor_resolved_ts_fallback.t3 exists cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^2$' 2 run task successfully killing cdc1: 2562 cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused run task failed 1-th time, retry later check diff failed 3-th time, retry later ERROR 1146 (42S02) at line 1: Table 'test.simple1' doesn't exist check data failed 5-th time, retry later Changefeed remove successfully. ID: changefeed-error CheckpointTs: 449575992876072964 SinkURI: file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-32148?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... check diff failed 3-th time, retry later check diff failed 2-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_basic/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:52:08 CST 2024] <<<<<< run test case canal_json_basic success! >>>>>> + set +x check_no_changefeed 127.0.0.1:2379 cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused parse error: Invalid numeric literal at line 1, column 6 run task successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8303 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8303 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8303 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:52:09 GMT < Content-Length: 1792 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66b56055-3a26-4577-bfbe-45c9145c65d6 {"id":"66b56055-3a26-4577-bfbe-45c9145c65d6","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996320} /tidb/cdc/default/__cdc_meta__/capture/dc6d139b-8be4-4c55-b3dd-ebb09c557025 {"id":"dc6d139b-8be4-4c55-b3dd-ebb09c557025","address":"127.0.0.1:8303","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996326} /tidb/cdc/default/__cdc_meta__/capture/f8c92dab-05fc-4d43-a50f-8f4048ae0ddb {"id":"f8c92dab-05fc-4d43-a50f-8f4048ae0ddb","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996323} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbecb8f8f 66b56055-3a26-4577-bfbe-45c9145c65d6 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbecb8fc4 f8c92dab-05fc-4d43-a50f-8f4048ae0ddb /tidb/cdc/default/__cdc_meta__/owner/22318f4dbecb8fe5 dc6d139b-8be4-4c55-b3dd-ebb09c557025 /tidb/cdc/default/default/upstream/7365853026311312504 {"id":7365853026311312504,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66b56055-3a26-4577-bfbe-45c9145c65d6 {"id":"66b56055-3a26-4577-bfbe-45c9145c65d6","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996320} /tidb/cdc/default/__cdc_meta__/capture/dc6d139b-8be4-4c55-b3dd-ebb09c557025 {"id":"dc6d139b-8be4-4c55-b3dd-ebb09c557025","address":"127.0.0.1:8303","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996326} /tidb/cdc/default/__cdc_meta__/capture/f8c92dab-05fc-4d43-a50f-8f4048ae0ddb {"id":"f8c92dab-05fc-4d43-a50f-8f4048ae0ddb","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996323} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbecb8f8f 66b56055-3a26-4577-bfbe-45c9145c65d6 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbecb8fc4 f8c92dab-05fc-4d43-a50f-8f4048ae0ddb /tidb/cdc/default/__cdc_meta__/owner/22318f4dbecb8fe5 dc6d139b-8be4-4c55-b3dd-ebb09c557025 /tidb/cdc/default/default/upstream/7365853026311312504 {"id":7365853026311312504,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66b56055-3a26-4577-bfbe-45c9145c65d6 {"id":"66b56055-3a26-4577-bfbe-45c9145c65d6","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996320} /tidb/cdc/default/__cdc_meta__/capture/dc6d139b-8be4-4c55-b3dd-ebb09c557025 {"id":"dc6d139b-8be4-4c55-b3dd-ebb09c557025","address":"127.0.0.1:8303","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996326} /tidb/cdc/default/__cdc_meta__/capture/f8c92dab-05fc-4d43-a50f-8f4048ae0ddb {"id":"f8c92dab-05fc-4d43-a50f-8f4048ae0ddb","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996323} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbecb8f8f 66b56055-3a26-4577-bfbe-45c9145c65d6 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbecb8fc4 f8c92dab-05fc-4d43-a50f-8f4048ae0ddb /tidb/cdc/default/__cdc_meta__/owner/22318f4dbecb8fe5 dc6d139b-8be4-4c55-b3dd-ebb09c557025 /tidb/cdc/default/default/upstream/7365853026311312504 + grep -q 'etcd info' {"id":7365853026311312504,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.cli.8887.out cli changefeed create --start-ts=449575994678837249 '--sink-uri=file:///tmp/tidb_cdc_test/multi_capture/storage_test/ticdc-multi-capture-test-23363?protocol=canal-json&enable-tidb-extension=true' --server=127.0.0.1:8301 Create changefeed successfully! ID: 8a2995ec-224b-4386-99e2-e261c9dac5e4 Info: {"upstream_id":7365853026311312504,"namespace":"default","id":"8a2995ec-224b-4386-99e2-e261c9dac5e4","sink_uri":"file:///tmp/tidb_cdc_test/multi_capture/storage_test/ticdc-multi-capture-test-23363?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:52:09.547015796+08:00","start_ts":449575994678837249,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575994678837249,"checkpoint_ts":449575994678837249,"checkpoint_time":"2024-05-06 19:51:57.592"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... run task failed 2-th time, retry later check diff failed 4-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 6-th time, retry later wait process cdc.test exit for 1-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/region_merge/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... check diff failed 4-th time, retry later go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/pingcap/tipb v0.0.0-20240318032315-55a7867ddd50 go: downloading github.com/danjacques/gofslock v0.0.0-20240212154529-d899e02bfe22 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/coocood/freecache v1.2.1 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/stretchr/testify v1.9.0 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/shirou/gopsutil/v3 v3.24.2 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/otiai10/copy v1.2.0 go: downloading golang.org/x/time v0.5.0 go: downloading github.com/aws/aws-sdk-go v1.50.0 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.1 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading github.com/klauspost/compress v1.17.8 go: downloading github.com/spf13/pflag v1.0.5 go: downloading golang.org/x/oauth2 v0.18.0 go: downloading google.golang.org/api v0.170.0 go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.0 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/fatih/color v1.16.0 go: downloading github.com/vbauerster/mpb/v7 v7.5.3 go: downloading golang.org/x/term v0.19.0 go: downloading github.com/spf13/cobra v1.8.0 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/google/pprof v0.0.0-20240117000934-35fc243c5815 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0 go: downloading github.com/gorilla/mux v1.8.0 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading golang.org/x/tools v0.20.0 go: downloading github.com/scalalang2/golang-fifo v0.1.5 go: downloading github.com/tidwall/btree v1.7.0 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/Masterminds/semver v1.5.0 go: downloading k8s.io/api v0.28.6 go: downloading github.com/emirpasic/gods v1.18.1 go: downloading github.com/pingcap/badger v1.5.1-0.20230103063557-828f39b09b6d go: downloading github.com/jfcg/sixb v1.3.8 go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d go: downloading github.com/mattn/go-runewidth v0.0.15 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading cloud.google.com/go v0.112.2 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading gopkg.in/yaml.v3 v3.0.1 wait process cdc.test exit for 2-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/multi_source/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:52:10 CST 2024] <<<<<< run test case multi_source success! >>>>>> go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.1 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.1 go: downloading golang.org/x/crypto v0.22.0 go: downloading cloud.google.com/go/compute/metadata v0.2.3 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading cloud.google.com/go/compute v1.25.1 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/kr/pretty v0.3.1 wait process cdc.test exit for 3-th time... + set +x + workdir=/tmp/tidb_cdc_test/multi_capture + sink_uri='file:///tmp/tidb_cdc_test/multi_capture/storage_test/ticdc-multi-capture-test-23363?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/multi_capture ++ date + echo '[Mon May 6 19:52:11 CST 2024] <<<<<< START storage consumer in multi_capture case >>>>>>' [Mon May 6 19:52:11 CST 2024] <<<<<< START storage consumer in multi_capture case >>>>>> + cd /tmp/tidb_cdc_test/multi_capture + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/multi_capture + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/multi_capture/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/multi_capture/storage_test/ticdc-multi-capture-test-23363?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table multi_capture_1.usertable not exists for 1-th check, retry later go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/golang-jwt/jwt/v5 v5.2.0 check diff failed 3-th time, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Mon May 6 19:52:11 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/InjectChangefeedDDLError=return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.60416043.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 go: downloading github.com/rogpeppe/go-internal v1.12.0 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/golang-jwt/jwt v3.2.2+incompatible =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_content_compatible/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:52:11 CST 2024] <<<<<< run test case canal_json_content_compatible success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fd06700015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-vs88z, pid:5145, start at 2024-05-06 19:52:10.938382849 +0800 CST m=+5.591961417 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:10.946 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:10.908 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:10.908 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading github.com/go-logr/logr v1.4.1 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 go: downloading github.com/DataDog/zstd v1.5.5 go: downloading k8s.io/apimachinery v0.28.6 check diff failed 5-th time, retry later check data successfully =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/processor_stop_delay/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... wait process cdc.test exit for 1-th time... go: downloading k8s.io/klog/v2 v2.120.1 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading github.com/google/gofuzz v1.2.0 go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... check diff failed 4-th time, retry later wait process cdc.test exit for 3-th time... table multi_capture_1.usertable not exists for 2-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fd06700015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-vs88z, pid:5145, start at 2024-05-06 19:52:10.938382849 +0800 CST m=+5.591961417 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:10.946 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:10.908 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:10.908 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fd065c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-vs88z, pid:5224, start at 2024-05-06 19:52:10.926140759 +0800 CST m=+5.502133188 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:10.933 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:10.903 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:10.903 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/csv_storage_basic/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/csv_storage_basic/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/csv_storage_basic/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/csv_storage_basic/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/csv_storage_basic/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 2-th time... start tidb cluster in /tmp/tidb_cdc_test/region_merge Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused run task failed 3-th time, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:52:14 CST 2024] <<<<<< run test case simple success! >>>>>> \033[0;36m<<< Run all test success >>>\033[0m check diff successfully [Mon May 6 19:52:14 CST 2024] <<<<<< START cdc server in owner_resign case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerDontUpdateCheckpoint=return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.30963098.out server --log-file /tmp/tidb_cdc_test/owner_resign/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/owner_resign/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 3-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:52:14 GMT < Content-Length: 897 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/587e09d5-bcc8-494e-9597-8d66a4fd6faf {"id":"587e09d5-bcc8-494e-9597-8d66a4fd6faf","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996331} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe6165ce 587e09d5-bcc8-494e-9597-8d66a4fd6faf /tidb/cdc/default/default/upstream/7365852910125063100 {"id":7365852910125063100,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/587e09d5-bcc8-494e-9597-8d66a4fd6faf {"id":"587e09d5-bcc8-494e-9597-8d66a4fd6faf","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996331} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe6165ce 587e09d5-bcc8-494e-9597-8d66a4fd6faf /tidb/cdc/default/default/upstream/7365852910125063100 {"id":7365852910125063100,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/587e09d5-bcc8-494e-9597-8d66a4fd6faf {"id":"587e09d5-bcc8-494e-9597-8d66a4fd6faf","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996331} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe6165ce 587e09d5-bcc8-494e-9597-8d66a4fd6faf /tidb/cdc/default/default/upstream/7365852910125063100 {"id":7365852910125063100,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6098.out cli changefeed create --start-ts=449575987528335361 '--sink-uri=file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-32148?protocol=canal-json&enable-tidb-extension=true' -c changefeed-error-1 cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:52:14 CST 2024] <<<<<< run test case capture_session_done_during_task success! >>>>>> [Pipeline] } Create changefeed successfully! ID: changefeed-error-1 Info: {"upstream_id":7365852910125063100,"namespace":"default","id":"changefeed-error-1","sink_uri":"file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-32148?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:52:15.095978402+08:00","start_ts":449575987528335361,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575987528335361,"checkpoint_ts":449575987528335361,"checkpoint_time":"2024-05-06 19:51:30.315"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... check diff failed 5-th time, retry later table multi_capture_1.usertable not exists for 3-th check, retry later Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } start tidb cluster in /tmp/tidb_cdc_test/processor_stop_delay Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... [Mon May 6 19:52:15 CST 2024] <<<<<< START cdc server in csv_storage_basic case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_basic.64696471.out server --log-file /tmp/tidb_cdc_test/csv_storage_basic/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/csv_storage_basic/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] // timeout [Pipeline] } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/multi_rocks/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } + set +x check_changefeed_status 127.0.0.1:8300 changefeed-error-1 warning last_warning ErrExecDDLFailed + endpoint=127.0.0.1:8300 + changefeed_id=changefeed-error-1 + expected_state=warning + field=last_warning + error_pattern=ErrExecDDLFailed ++ curl 127.0.0.1:8300/api/v2/changefeeds/changefeed-error-1/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading github.com/google/s2a-go v0.1.7 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading github.com/felixge/httpsnoop v1.0.4 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 253 100 253 0 0 2294 0 --:--:-- --:--:-- --:--:-- 2300 + info='{"state":"warning","resolved_ts":449575987921289235,"checkpoint_ts":449575987921289235,"last_warning":{"time":"2024-05-06T19:52:16.988218015+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}}' + echo '{"state":"warning","resolved_ts":449575987921289235,"checkpoint_ts":449575987921289235,"last_warning":{"time":"2024-05-06T19:52:16.988218015+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}}' {"state":"warning","resolved_ts":449575987921289235,"checkpoint_ts":449575987921289235,"last_warning":{"time":"2024-05-06T19:52:16.988218015+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}} ++ jq -r .state ++ echo '{"state":"warning","resolved_ts":449575987921289235,"checkpoint_ts":449575987921289235,"last_warning":{"time":"2024-05-06T19:52:16.988218015+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec' DDL 'failed"}}' + state=warning + [[ ! warning == \w\a\r\n\i\n\g ]] + [[ -z last_warning ]] ++ echo '{"state":"warning","resolved_ts":449575987921289235,"checkpoint_ts":449575987921289235,"last_warning":{"time":"2024-05-06T19:52:16.988218015+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec' DDL 'failed"}}' ++ jq -r .last_warning.message + error_msg='[CDC:ErrExecDDLFailed]exec DDL failed' + [[ ! [CDC:ErrExecDDLFailed]exec DDL failed =~ ErrExecDDLFailed ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6150.out cli changefeed remove -c changefeed-error-1 table multi_capture_1.usertable not exists for 4-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:52:17 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/d76b659f-2a16-4cbe-89f9-8173fe4be567 {UpstreamID:7365852992115240155 Namespace:default ID:d76b659f-2a16-4cbe-89f9-8173fe4be567 SinkURI:file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-8883?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:51:50.739873053 +0800 CST StartTs:449575992855363586 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00321c090 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575992881577985} {CheckpointTs:449575999448285192 MinTableBarrierTs:449575999710429193 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7ec7e3cf-a880-47bd-9958-2b2800e887ea {"id":"7ec7e3cf-a880-47bd-9958-2b2800e887ea","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996334} /tidb/cdc/default/__cdc_meta__/capture/a8c96bb4-88b2-4a96-a776-a9800b328e81 {"id":"a8c96bb4-88b2-4a96-a776-a9800b328e81","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996307} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbea63a32 a8c96bb4-88b2-4a96-a776-a9800b328e81 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbea63b6f 7ec7e3cf-a880-47bd-9958-2b2800e887ea /tidb/cdc/default/default/changefeed/info/d76b659f-2a16-4cbe-89f9-8173fe4be567 {"upstream-id":7365852992115240155,"namespace":"default","changefeed-id":"d76b659f-2a16-4cbe-89f9-8173fe4be567","sink-uri":"file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-8883?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:51:50.739873053+08:00","start-ts":449575992855363586,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575992881577985} /tidb/cdc/default/default/changefeed/status/d76b659f-2a16-4cbe-89f9-8173fe4be567 {"checkpoint-ts":449575999448285192,"min-table-barrier-ts":449575999710429193,"admin-job-type":0} /tidb/cdc/default/default/task/position/7ec7e3cf-a880-47bd-9958-2b2800e887ea/d76b659f-2a16-4cbe-89f9-8173fe4be567 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/a8c96bb4-88b2-4a96-a776-a9800b328e81/d76b659f-2a16-4cbe-89f9-8173fe4be567 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852992115240155 {"id":7365852992115240155,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/d76b659f-2a16-4cbe-89f9-8173fe4be567 {UpstreamID:7365852992115240155 Namespace:default ID:d76b659f-2a16-4cbe-89f9-8173fe4be567 SinkURI:file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-8883?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:51:50.739873053 +0800 CST StartTs:449575992855363586 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00321c090 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575992881577985} {CheckpointTs:449575999448285192 MinTableBarrierTs:449575999710429193 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7ec7e3cf-a880-47bd-9958-2b2800e887ea {"id":"7ec7e3cf-a880-47bd-9958-2b2800e887ea","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996334} /tidb/cdc/default/__cdc_meta__/capture/a8c96bb4-88b2-4a96-a776-a9800b328e81 {"id":"a8c96bb4-88b2-4a96-a776-a9800b328e81","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996307} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbea63a32 a8c96bb4-88b2-4a96-a776-a9800b328e81 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbea63b6f 7ec7e3cf-a880-47bd-9958-2b2800e887ea /tidb/cdc/default/default/changefeed/info/d76b659f-2a16-4cbe-89f9-8173fe4be567 {"upstream-id":7365852992115240155,"namespace":"default","changefeed-id":"d76b659f-2a16-4cbe-89f9-8173fe4be567","sink-uri":"file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-8883?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:51:50.739873053+08:00","start-ts":449575992855363586,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575992881577985} /tidb/cdc/default/default/changefeed/status/d76b659f-2a16-4cbe-89f9-8173fe4be567 {"checkpoint-ts":449575999448285192,"min-table-barrier-ts":449575999710429193,"admin-job-type":0} /tidb/cdc/default/default/task/position/7ec7e3cf-a880-47bd-9958-2b2800e887ea/d76b659f-2a16-4cbe-89f9-8173fe4be567 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/a8c96bb4-88b2-4a96-a776-a9800b328e81/d76b659f-2a16-4cbe-89f9-8173fe4be567 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852992115240155 {"id":7365852992115240155,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/d76b659f-2a16-4cbe-89f9-8173fe4be567 {UpstreamID:7365852992115240155 Namespace:default ID:d76b659f-2a16-4cbe-89f9-8173fe4be567 SinkURI:file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-8883?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:51:50.739873053 +0800 CST StartTs:449575992855363586 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00321c090 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575992881577985} {CheckpointTs:449575999448285192 MinTableBarrierTs:449575999710429193 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7ec7e3cf-a880-47bd-9958-2b2800e887ea {"id":"7ec7e3cf-a880-47bd-9958-2b2800e887ea","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996334} /tidb/cdc/default/__cdc_meta__/capture/a8c96bb4-88b2-4a96-a776-a9800b328e81 {"id":"a8c96bb4-88b2-4a96-a776-a9800b328e81","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996307} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbea63a32 a8c96bb4-88b2-4a96-a776-a9800b328e81 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbea63b6f 7ec7e3cf-a880-47bd-9958-2b2800e887ea /tidb/cdc/default/default/changefeed/info/d76b659f-2a16-4cbe-89f9-8173fe4be567 {"upstream-id":7365852992115240155,"namespace":"default","changefeed-id":"d76b659f-2a16-4cbe-89f9-8173fe4be567","sink-uri":"file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-8883?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:51:50.739873053+08:00","start-ts":449575992855363586,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575992881577985} /tidb/cdc/default/default/changefeed/status/d76b659f-2a16-4cbe-89f9-8173fe4be567 {"checkpoint-ts":449575999448285192,"min-table-barrier-ts":449575999710429193,"admin-job-type":0} /tidb/cdc/default/default/task/position/7ec7e3cf-a880-47bd-9958-2b2800e887ea/d76b659f-2a16-4cbe-89f9-8173fe4be567 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/a8c96bb4-88b2-4a96-a776-a9800b328e81/d76b659f-2a16-4cbe-89f9-8173fe4be567 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852992115240155 {"id":7365852992115240155,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^2$' 2 run task successfully % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 go: downloading github.com/jmespath/go-jmespath v0.4.0 check diff failed 6-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/multi_topics/run.sh using Sink-Type: storage... <<================= Changefeed remove successfully. [Mon May 6 19:52:18 CST 2024] <<<<<< run test case multi_topics success! >>>>>> ID: changefeed-error-1 CheckpointTs: 449575987921289235 SinkURI: file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-32148?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table multi_capture_1.usertable not exists for 5-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:52:18 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/809d7195-f447-40ae-b77d-f3e1b53fa2ad {"id":"809d7195-f447-40ae-b77d-f3e1b53fa2ad","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996336} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbf141332 809d7195-f447-40ae-b77d-f3e1b53fa2ad /tidb/cdc/default/default/upstream/7365853116143173555 {"id":7365853116143173555,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/809d7195-f447-40ae-b77d-f3e1b53fa2ad {"id":"809d7195-f447-40ae-b77d-f3e1b53fa2ad","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996336} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbf141332 809d7195-f447-40ae-b77d-f3e1b53fa2ad /tidb/cdc/default/default/upstream/7365853116143173555 {"id":7365853116143173555,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/809d7195-f447-40ae-b77d-f3e1b53fa2ad {"id":"809d7195-f447-40ae-b77d-f3e1b53fa2ad","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996336} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbf141332 809d7195-f447-40ae-b77d-f3e1b53fa2ad /tidb/cdc/default/default/upstream/7365853116143173555 {"id":7365853116143173555,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_basic.cli.6526.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/csv_storage_basic/storage_test?flush-interval=5s' --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_basic/conf/changefeed.toml Create changefeed successfully! ID: 2944f63d-4523-4212-a1c3-6526d773711c Info: {"upstream_id":7365853116143173555,"namespace":"default","id":"2944f63d-4523-4212-a1c3-6526d773711c","sink_uri":"file:///tmp/tidb_cdc_test/csv_storage_basic/storage_test?flush-interval=5s","create_time":"2024-05-06T19:52:19.323565107+08:00","start_ts":449576000358449153,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576000358449153,"checkpoint_ts":449576000358449153,"checkpoint_time":"2024-05-06 19:52:19.258"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... start tidb cluster in /tmp/tidb_cdc_test/multi_rocks Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... + set +x wait process cdc.test exit for 1-th time... cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' 1 run task successfully check diff failed 1-th time, retry later wait process cdc.test exit for 2-th time... + set +x check diff failed 1-th time, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table multi_capture_1.usertable not exists for 6-th check, retry later check diff failed 7-th time, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Mon May 6 19:52:21 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/txnutil/gc/InjectActualGCSafePoint=return(9223372036854775807)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.62086210.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + workdir=/tmp/tidb_cdc_test/csv_storage_basic + sink_uri='file:///tmp/tidb_cdc_test/csv_storage_basic/storage_test?flush-interval=5s' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_basic/conf/changefeed.toml + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/csv_storage_basic ++ date + echo '[Mon May 6 19:52:21 CST 2024] <<<<<< START storage consumer in csv_storage_basic case >>>>>>' [Mon May 6 19:52:21 CST 2024] <<<<<< START storage consumer in csv_storage_basic case >>>>>> + cd /tmp/tidb_cdc_test/csv_storage_basic + '[' /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_basic/conf/changefeed.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/csv_storage_basic + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/csv_storage_basic/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/csv_storage_basic/storage_test?flush-interval=5s' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_basic/conf/changefeed.toml Verifying downstream PD is started... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/avro_basic/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:52:21 CST 2024] <<<<<< run test case avro_basic success! >>>>>> check diff failed 2-th time, retry later check diff failed 2-th time, retry later Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Debugger for raftstore-v2 is used Debugger for raftstore-v2 is used Debugger for raftstore-v2 is used Debugger for raftstore-v2 is used table multi_capture_1.usertable exists table multi_capture_2.usertable exists table multi_capture_3.usertable exists table multi_capture_4.usertable not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 3-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:52:24 GMT < Content-Length: 897 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/51c58e9d-fc9b-4b65-813d-a623fa3994e2 {"id":"51c58e9d-fc9b-4b65-813d-a623fa3994e2","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996341} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe616646 51c58e9d-fc9b-4b65-813d-a623fa3994e2 /tidb/cdc/default/default/upstream/7365852910125063100 {"id":7365852910125063100,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/51c58e9d-fc9b-4b65-813d-a623fa3994e2 {"id":"51c58e9d-fc9b-4b65-813d-a623fa3994e2","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996341} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe616646 51c58e9d-fc9b-4b65-813d-a623fa3994e2 /tidb/cdc/default/default/upstream/7365852910125063100 {"id":7365852910125063100,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/51c58e9d-fc9b-4b65-813d-a623fa3994e2 {"id":"51c58e9d-fc9b-4b65-813d-a623fa3994e2","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996341} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe616646 51c58e9d-fc9b-4b65-813d-a623fa3994e2 /tidb/cdc/default/default/upstream/7365852910125063100 {"id":7365852910125063100,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6264.out cli changefeed create --start-ts=449575987528335361 '--sink-uri=file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-32148?protocol=canal-json&enable-tidb-extension=true' -c changefeed-error-2 Create changefeed successfully! ID: changefeed-error-2 Info: {"upstream_id":7365852910125063100,"namespace":"default","id":"changefeed-error-2","sink_uri":"file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-32148?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:52:24.564493596+08:00","start_ts":449575987528335361,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575987528335361,"checkpoint_ts":449575987528335361,"checkpoint_time":"2024-05-06 19:51:30.315"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_handle_key_only/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:52:24 CST 2024] <<<<<< run test case canal_json_handle_key_only success! >>>>>> check diff failed 1-th time, retry later check diff failed 3-th time, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table multi_capture_4.usertable not exists for 2-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-error-2 failed [CDC:ErrSnapshotLostByGC] + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error-2 + expected_state=failed + error_msg='[CDC:ErrSnapshotLostByGC]' + tls_dir='[CDC:ErrSnapshotLostByGC]' + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error-2 -s =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/cdc_server_tips/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... + info='{ "upstream_id": 7365852910125063100, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 449575987528335361, "checkpoint_time": "2024-05-06 19:51:30.315", "error": { "time": "2024-05-06T19:52:24.656452329+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449575987528335361 is earlier than or equal to GC safepoint at 9223372036854775807" } }' + echo '{ "upstream_id": 7365852910125063100, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 449575987528335361, "checkpoint_time": "2024-05-06 19:51:30.315", "error": { "time": "2024-05-06T19:52:24.656452329+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449575987528335361 is earlier than or equal to GC safepoint at 9223372036854775807" } }' { "upstream_id": 7365852910125063100, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 449575987528335361, "checkpoint_time": "2024-05-06 19:51:30.315", "error": { "time": "2024-05-06T19:52:24.656452329+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449575987528335361 is earlier than or equal to GC safepoint at 9223372036854775807" } } ++ echo '{' '"upstream_id":' 7365852910125063100, '"namespace":' '"default",' '"id":' '"changefeed-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 449575987528335361, '"checkpoint_time":' '"2024-05-06' '19:51:30.315",' '"error":' '{' '"time":' '"2024-05-06T19:52:24.656452329+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSnapshotLostByGC",' '"message":' '"[CDC:ErrSnapshotLostByGC]fail' to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449575987528335361 is earlier than or equal to GC safepoint at '9223372036854775807"' '}' '}' ++ jq -r .state + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ jq -r .error.message ++ echo '{' '"upstream_id":' 7365852910125063100, '"namespace":' '"default",' '"id":' '"changefeed-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 449575987528335361, '"checkpoint_time":' '"2024-05-06' '19:51:30.315",' '"error":' '{' '"time":' '"2024-05-06T19:52:24.656452329+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSnapshotLostByGC",' '"message":' '"[CDC:ErrSnapshotLostByGC]fail' to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449575987528335361 is earlier than or equal to GC safepoint at '9223372036854775807"' '}' '}' + message='[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449575987528335361 is earlier than or equal to GC safepoint at 9223372036854775807' + [[ ! [CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449575987528335361 is earlier than or equal to GC safepoint at 9223372036854775807 =~ \[CDC:ErrSnapshotLostByGC] ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6345.out cli changefeed remove -c changefeed-error-2 check diff successfully wait process cdc.test exit for 1-th time... check diff failed 4-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/ddl_attributes/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table multi_capture_4.usertable not exists for 3-th check, retry later Changefeed remove successfully. ID: changefeed-error-2 CheckpointTs: 449575987528335361 SinkURI: file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-32148?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fdf6340013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-dscp5, pid:3964, start at 2024-05-06 19:52:26.282956107 +0800 CST m=+5.419825247 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:26.290 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:26.253 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:26.253 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fdfd08000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-t7zm9, pid:6510, start at 2024-05-06 19:52:26.708812585 +0800 CST m=+5.552414816 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:26.716 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:26.690 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:26.690 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 3-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/open_protocol_handle_key_only/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:52:27 CST 2024] <<<<<< run test case open_protocol_handle_key_only success! >>>>>> cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:52:28 CST 2024] <<<<<< run test case processor_resolved_ts_fallback success! >>>>>> + set +x check diff failed 5-th time, retry later table multi_capture_4.usertable not exists for 4-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fdf6340013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-dscp5, pid:3964, start at 2024-05-06 19:52:26.282956107 +0800 CST m=+5.419825247 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:26.290 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:26.253 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:26.253 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fdf7b00012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-dscp5, pid:4046, start at 2024-05-06 19:52:26.366479342 +0800 CST m=+5.450924351 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:26.372 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:26.348 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:26.348 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/region_merge/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/region_merge/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 1-th time... check diff failed 1-th time, retry later check diff failed 3-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fdfd08000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-t7zm9, pid:6510, start at 2024-05-06 19:52:26.708812585 +0800 CST m=+5.552414816 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:26.716 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:26.690 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:26.690 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fdfe900005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-t7zm9, pid:6586, start at 2024-05-06 19:52:26.79224214 +0800 CST m=+5.566063035 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:26.799 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:26.788 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:26.788 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Mon May 6 19:52:30 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/redo/ChangefeedNewRedoManagerError=2*return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.63866388.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 start tidb cluster in /tmp/tidb_cdc_test/ddl_attributes Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... check diff failed 2-th time, retry later check diff failed 6-th time, retry later [Mon May 6 19:52:31 CST 2024] <<<<<< START cdc server in processor_stop_delay case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorStopDelay=1*sleep(10000)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_stop_delay.79077909.out server --log-file /tmp/tidb_cdc_test/processor_stop_delay/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_stop_delay/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table multi_capture_4.usertable not exists for 5-th check, retry later [Mon May 6 19:52:31 CST 2024] <<<<<< START cdc server in region_merge case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.region_merge.53505352.out server --log-file /tmp/tidb_cdc_test/region_merge/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/region_merge/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 4-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_claim_check/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:52:32 CST 2024] <<<<<< run test case canal_json_claim_check success! >>>>>> check diff successfully % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table multi_capture_4.usertable exists check diff successfully ***************** properties ***************** "requestdistribution"="uniform" "readallfields"="true" "workload"="core" "mysql.port"="4000" "readproportion"="0" "mysql.host"="127.0.0.1" "insertproportion"="0" "threadcount"="2" "dotransactions"="false" "mysql.user"="root" "mysql.db"="multi_capture_1" "scanproportion"="0" "recordcount"="20" "updateproportion"="0" "operationcount"="0" ********************************************** Run finished, takes 17.212859ms INSERT - Takes(s): 0.0, Count: 20, OPS: 1340.2, Avg(us): 1667, Min(us): 589, Max(us): 5026, 95th(us): 5000, 99th(us): 6000 ***************** properties ***************** "threadcount"="2" "readallfields"="true" "workload"="core" "mysql.port"="4000" "mysql.db"="multi_capture_2" "insertproportion"="0" "dotransactions"="false" "readproportion"="0" "requestdistribution"="uniform" "operationcount"="0" "scanproportion"="0" "recordcount"="20" "mysql.host"="127.0.0.1" "mysql.user"="root" "updateproportion"="0" ********************************************** Run finished, takes 13.096475ms INSERT - Takes(s): 0.0, Count: 20, OPS: 1799.4, Avg(us): 1262, Min(us): 561, Max(us): 2231, 95th(us): 2000, 99th(us): 3000 ***************** properties ***************** "mysql.user"="root" "requestdistribution"="uniform" "dotransactions"="false" "updateproportion"="0" "mysql.db"="multi_capture_3" "recordcount"="20" "operationcount"="0" "mysql.port"="4000" "threadcount"="2" "workload"="core" "readallfields"="true" "readproportion"="0" "insertproportion"="0" "mysql.host"="127.0.0.1" "scanproportion"="0" ********************************************** Run finished, takes 14.883288ms INSERT - Takes(s): 0.0, Count: 20, OPS: 1505.2, Avg(us): 1332, Min(us): 705, Max(us): 2914, 95th(us): 3000, 99th(us): 3000 ***************** properties ***************** "mysql.db"="multi_capture_4" "mysql.host"="127.0.0.1" "operationcount"="0" "insertproportion"="0" "readallfields"="true" "dotransactions"="false" "mysql.user"="root" "workload"="core" "mysql.port"="4000" "updateproportion"="0" "scanproportion"="0" "recordcount"="20" "threadcount"="2" "requestdistribution"="uniform" "readproportion"="0" ********************************************** Run finished, takes 11.198435ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2007.7, Avg(us): 1065, Min(us): 531, Max(us): 2003, 95th(us): 2000, 99th(us): 3000 check diff failed 1-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff successfully [Mon May 6 19:52:33 CST 2024] <<<<<< run test case csv_storage_basic success! >>>>>> check diff failed 5-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:52:34 GMT < Content-Length: 873 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/17a43921-4954-4088-83fe-689c5d3a238a {"id":"17a43921-4954-4088-83fe-689c5d3a238a","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996351} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbf528c70 17a43921-4954-4088-83fe-689c5d3a238a /tidb/cdc/default/default/upstream/7365853181280487954 {"id":7365853181280487954,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/17a43921-4954-4088-83fe-689c5d3a238a {"id":"17a43921-4954-4088-83fe-689c5d3a238a","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996351} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbf528c70 17a43921-4954-4088-83fe-689c5d3a238a /tidb/cdc/default/default/upstream/7365853181280487954 {"id":7365853181280487954,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/17a43921-4954-4088-83fe-689c5d3a238a {"id":"17a43921-4954-4088-83fe-689c5d3a238a","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996351} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbf528c70 17a43921-4954-4088-83fe-689c5d3a238a /tidb/cdc/default/default/upstream/7365853181280487954 {"id":7365853181280487954,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: b3a3b6c4-7f03-4975-bf33-a1f719b564dc Info: {"upstream_id":7365853181280487954,"namespace":"default","id":"b3a3b6c4-7f03-4975-bf33-a1f719b564dc","sink_uri":"file:///tmp/tidb_cdc_test/region_merge/storage_test/ticdc-region-merge-test-30342?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:52:34.435793829+08:00","start_ts":449576004302667782,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576004302667782,"checkpoint_ts":449576004302667782,"checkpoint_time":"2024-05-06 19:52:34.304"} + workdir=/tmp/tidb_cdc_test/region_merge + sink_uri='file:///tmp/tidb_cdc_test/region_merge/storage_test/ticdc-region-merge-test-30342?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/region_merge ++ date + echo '[Mon May 6 19:52:34 CST 2024] <<<<<< START storage consumer in region_merge case >>>>>>' [Mon May 6 19:52:34 CST 2024] <<<<<< START storage consumer in region_merge case >>>>>> + cd /tmp/tidb_cdc_test/region_merge + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/region_merge + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/region_merge/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/region_merge/storage_test/ticdc-region-merge-test-30342?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' split_and_random_merge scale: 20 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:52:34 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d5ce9edc-4419-4460-8e93-f2b45ef4c16a {"id":"d5ce9edc-4419-4460-8e93-f2b45ef4c16a","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996351} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbf556f2f d5ce9edc-4419-4460-8e93-f2b45ef4c16a /tidb/cdc/default/default/upstream/7365853182811236687 {"id":7365853182811236687,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d5ce9edc-4419-4460-8e93-f2b45ef4c16a {"id":"d5ce9edc-4419-4460-8e93-f2b45ef4c16a","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996351} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbf556f2f d5ce9edc-4419-4460-8e93-f2b45ef4c16a /tidb/cdc/default/default/upstream/7365853182811236687 {"id":7365853182811236687,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d5ce9edc-4419-4460-8e93-f2b45ef4c16a {"id":"d5ce9edc-4419-4460-8e93-f2b45ef4c16a","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996351} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbf556f2f d5ce9edc-4419-4460-8e93-f2b45ef4c16a /tidb/cdc/default/default/upstream/7365853182811236687 {"id":7365853182811236687,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + workdir=/tmp/tidb_cdc_test/processor_stop_delay + sink_uri='file:///tmp/tidb_cdc_test/processor_stop_delay/storage_test/ticdc-processor-stop-delay-14002?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/processor_stop_delay ++ date + echo '[Mon May 6 19:52:34 CST 2024] <<<<<< START storage consumer in processor_stop_delay case >>>>>>' [Mon May 6 19:52:34 CST 2024] <<<<<< START storage consumer in processor_stop_delay case >>>>>> + cd /tmp/tidb_cdc_test/processor_stop_delay + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/processor_stop_delay + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/processor_stop_delay/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/processor_stop_delay/storage_test/ticdc-processor-stop-delay-14002?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table processor_stop_delay.t not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:52:34 GMT < Content-Length: 897 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/48417fac-928b-4bc8-b7b6-724ac4e85347 {"id":"48417fac-928b-4bc8-b7b6-724ac4e85347","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996350} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 7 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe6166b6 48417fac-928b-4bc8-b7b6-724ac4e85347 /tidb/cdc/default/default/upstream/7365852910125063100 {"id":7365852910125063100,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/48417fac-928b-4bc8-b7b6-724ac4e85347 {"id":"48417fac-928b-4bc8-b7b6-724ac4e85347","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996350} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 7 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe6166b6 48417fac-928b-4bc8-b7b6-724ac4e85347 /tidb/cdc/default/default/upstream/7365852910125063100 {"id":7365852910125063100,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/48417fac-928b-4bc8-b7b6-724ac4e85347 {"id":"48417fac-928b-4bc8-b7b6-724ac4e85347","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996350} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 7 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe6166b6 48417fac-928b-4bc8-b7b6-724ac4e85347 /tidb/cdc/default/default/upstream/7365852910125063100 {"id":7365852910125063100,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6442.out cli changefeed create --start-ts=0 '--sink-uri=file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-32148?protocol=canal-json&enable-tidb-extension=true' -c changefeed-initialize-error Create changefeed successfully! ID: changefeed-initialize-error Info: {"upstream_id":7365852910125063100,"namespace":"default","id":"changefeed-initialize-error","sink_uri":"file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-32148?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:52:35.093979064+08:00","start_ts":449576004475682820,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576004475682820,"checkpoint_ts":449576004475682820,"checkpoint_time":"2024-05-06 19:52:34.964"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... check diff failed 6-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/cdc_server_tips Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/open_protocol_claim_check/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:52:35 CST 2024] <<<<<< run test case open_protocol_claim_check success! >>>>>> check diff failed 2-th time, retry later + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7365852910125063100, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 449576004475682820, "checkpoint_time": "2024-05-06 19:52:34.964", "error": { "time": "2024-05-06T19:52:35.285114562+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' + echo '{ "upstream_id": 7365852910125063100, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 449576004475682820, "checkpoint_time": "2024-05-06 19:52:34.964", "error": { "time": "2024-05-06T19:52:35.285114562+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' { "upstream_id": 7365852910125063100, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 449576004475682820, "checkpoint_time": "2024-05-06 19:52:34.964", "error": { "time": "2024-05-06T19:52:35.285114562+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } } ++ echo '{' '"upstream_id":' 7365852910125063100, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"warning",' '"checkpoint_tso":' 449576004475682820, '"checkpoint_time":' '"2024-05-06' '19:52:34.964",' '"error":' '{' '"time":' '"2024-05-06T19:52:35.285114562+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .state + state=warning + [[ ! warning == \n\o\r\m\a\l ]] + echo 'changefeed state warning does not equal to normal' changefeed state warning does not equal to normal + exit 1 run task failed 1-th time, retry later Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table processor_stop_delay.t not exists for 2-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 7-th time, retry later check diff failed 3-th time, retry later check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7365852910125063100, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 449576004475682820, "checkpoint_time": "2024-05-06 19:52:34.964", "error": { "time": "2024-05-06T19:52:35.285114562+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' + echo '{ "upstream_id": 7365852910125063100, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 449576004475682820, "checkpoint_time": "2024-05-06 19:52:34.964", "error": { "time": "2024-05-06T19:52:35.285114562+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' { "upstream_id": 7365852910125063100, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 449576004475682820, "checkpoint_time": "2024-05-06 19:52:34.964", "error": { "time": "2024-05-06T19:52:35.285114562+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } } ++ echo '{' '"upstream_id":' 7365852910125063100, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"warning",' '"checkpoint_tso":' 449576004475682820, '"checkpoint_time":' '"2024-05-06' '19:52:34.964",' '"error":' '{' '"time":' '"2024-05-06T19:52:35.285114562+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .state + state=warning + [[ ! warning == \n\o\r\m\a\l ]] + echo 'changefeed state warning does not equal to normal' changefeed state warning does not equal to normal + exit 1 run task failed 2-th time, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_stop_delay.t not exists for 3-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_basic/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 8-th time, retry later check diff failed 4-th time, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } *************************** 1. row *************************** count(distinct region_id): 2 [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // container [Pipeline] } table processor_stop_delay.t not exists for 4-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fee1c8000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w7kft, pid:4305, start at 2024-05-06 19:52:41.343664248 +0800 CST m=+5.616947008 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:41.350 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:41.330 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:41.330 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fee1c8000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w7kft, pid:4305, start at 2024-05-06 19:52:41.343664248 +0800 CST m=+5.616947008 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:41.350 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:41.330 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:41.330 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fee41c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-w7kft, pid:4368, start at 2024-05-06 19:52:41.507818077 +0800 CST m=+5.711967861 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:41.515 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:41.479 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:41.479 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... check diff failed 9-th time, retry later [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_attributes/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_attributes/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Pipeline] } check diff successfully wait process cdc.test exit for 1-th time... check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s check diff successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_stop_delay.t not exists for 5-th check, retry later + info='{ "upstream_id": 7365852910125063100, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 449576006520406021, "checkpoint_time": "2024-05-06 19:52:42.764", "error": null }' + echo '{ "upstream_id": 7365852910125063100, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 449576006520406021, "checkpoint_time": "2024-05-06 19:52:42.764", "error": null }' { "upstream_id": 7365852910125063100, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 449576006520406021, "checkpoint_time": "2024-05-06 19:52:42.764", "error": null } ++ echo '{' '"upstream_id":' 7365852910125063100, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 449576006520406021, '"checkpoint_time":' '"2024-05-06' '19:52:42.764",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7365852910125063100, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 449576006520406021, '"checkpoint_time":' '"2024-05-06' '19:52:42.764",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6628.out cli changefeed pause -c changefeed-initialize-error ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... wait process cdc.test exit for 1-th time... check diff successfully PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... start tidb cluster in /tmp/tidb_cdc_test/canal_json_storage_basic Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.cli.5654.out cli tso query --pd=http://127.0.0.1:2379 wait process cdc.test exit for 3-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:52:44 CST 2024] <<<<<< run test case multi_capture success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:52:45 CST 2024] <<<<<< run test case owner_resign success! >>>>>> table processor_stop_delay.t exists check diff successfully + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error stopped changefeed new redo manager injected error + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=stopped + error_msg=changefeed + tls_dir=error + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) split_and_random_merge scale: 40 + info='{ "upstream_id": 7365852910125063100, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 449576006520406021, "checkpoint_time": "2024-05-06 19:52:42.764", "error": { "time": "2024-05-06T19:52:35.285114562+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' + echo '{ "upstream_id": 7365852910125063100, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 449576006520406021, "checkpoint_time": "2024-05-06 19:52:42.764", "error": { "time": "2024-05-06T19:52:35.285114562+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' { "upstream_id": 7365852910125063100, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 449576006520406021, "checkpoint_time": "2024-05-06 19:52:42.764", "error": { "time": "2024-05-06T19:52:35.285114562+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } } ++ echo '{' '"upstream_id":' 7365852910125063100, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"stopped",' '"checkpoint_tso":' 449576006520406021, '"checkpoint_time":' '"2024-05-06' '19:52:42.764",' '"error":' '{' '"time":' '"2024-05-06T19:52:35.285114562+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7365852910125063100, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"stopped",' '"checkpoint_tso":' 449576006520406021, '"checkpoint_time":' '"2024-05-06' '19:52:42.764",' '"error":' '{' '"time":' '"2024-05-06T19:52:35.285114562+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .error.message + message='changefeed new redo manager injected error' + [[ ! changefeed new redo manager injected error =~ changefeed ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6711.out cli changefeed resume -c changefeed-initialize-error check diff failed 1-th time, retry later + set +x + tso='449576006943768586 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576006943768586 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:52:45 CST 2024] <<<<<< START cdc server in ddl_attributes case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.56875689.out server --log-file /tmp/tidb_cdc_test/ddl_attributes/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_attributes/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 PASS coverage: 2.3% of statements in github.com/pingcap/tiflow/... check diff failed 1-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ff12b00017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3, pid:4055, start at 2024-05-06 19:52:44.48474386 +0800 CST m=+5.386752826 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:44.490 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:44.460 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:44.460 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ff12b00017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3, pid:4055, start at 2024-05-06 19:52:44.48474386 +0800 CST m=+5.386752826 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:44.490 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:44.460 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:44.460 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ff13900014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3, pid:4132, start at 2024-05-06 19:52:44.561632948 +0800 CST m=+5.414612613 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:44.571 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:44.567 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:44.567 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7365852910125063100, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 449576006520406021, "checkpoint_time": "2024-05-06 19:52:42.764", "error": null }' + echo '{ "upstream_id": 7365852910125063100, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 449576006520406021, "checkpoint_time": "2024-05-06 19:52:42.764", "error": null }' { "upstream_id": 7365852910125063100, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 449576006520406021, "checkpoint_time": "2024-05-06 19:52:42.764", "error": null } ++ echo '{' '"upstream_id":' 7365852910125063100, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 449576006520406021, '"checkpoint_time":' '"2024-05-06' '19:52:42.764",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7365852910125063100, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 449576006520406021, '"checkpoint_time":' '"2024-05-06' '19:52:42.764",' '"error":' null '}' ++ jq -r .error.message ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + message=null + [[ ! null =~ null ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6802.out cli changefeed remove -c changefeed-initialize-error Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff failed 2-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... Changefeed remove successfully. ID: changefeed-initialize-error CheckpointTs: 449576006520406021 SinkURI: file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-32148?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:52:48 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9edf937e-8ff4-4cac-9711-556f61d2380e {"id":"9edf937e-8ff4-4cac-9711-556f61d2380e","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996366} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbf8cad33 9edf937e-8ff4-4cac-9711-556f61d2380e /tidb/cdc/default/default/upstream/7365853247017871085 {"id":7365853247017871085,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9edf937e-8ff4-4cac-9711-556f61d2380e {"id":"9edf937e-8ff4-4cac-9711-556f61d2380e","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996366} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbf8cad33 9edf937e-8ff4-4cac-9711-556f61d2380e /tidb/cdc/default/default/upstream/7365853247017871085 {"id":7365853247017871085,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9edf937e-8ff4-4cac-9711-556f61d2380e {"id":"9edf937e-8ff4-4cac-9711-556f61d2380e","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996366} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbf8cad33 9edf937e-8ff4-4cac-9711-556f61d2380e /tidb/cdc/default/default/upstream/7365853247017871085 {"id":7365853247017871085,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.cli.5751.out cli changefeed create --start-ts=449576006943768586 '--sink-uri=file:///tmp/tidb_cdc_test/ddl_attributes/storage_test/ticdc-ddl-attributes-test-5389?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: 28eb471a-1ad9-470c-b1d9-d2101506ba3b Info: {"upstream_id":7365853247017871085,"namespace":"default","id":"28eb471a-1ad9-470c-b1d9-d2101506ba3b","sink_uri":"file:///tmp/tidb_cdc_test/ddl_attributes/storage_test/ticdc-ddl-attributes-test-5389?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:52:49.410200455+08:00","start_ts":449576006943768586,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576006943768586,"checkpoint_ts":449576006943768586,"checkpoint_time":"2024-05-06 19:52:44.379"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.cli.5393.out cli tso query --pd=http://127.0.0.1:2379 Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 3-th time, retry later + set +x check diff failed 2-th time, retry later wait process cdc.test exit for 1-th time... + set +x + workdir=/tmp/tidb_cdc_test/ddl_attributes + sink_uri='file:///tmp/tidb_cdc_test/ddl_attributes/storage_test/ticdc-ddl-attributes-test-5389?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/ddl_attributes ++ date + echo '[Mon May 6 19:52:50 CST 2024] <<<<<< START storage consumer in ddl_attributes case >>>>>>' [Mon May 6 19:52:50 CST 2024] <<<<<< START storage consumer in ddl_attributes case >>>>>> + cd /tmp/tidb_cdc_test/ddl_attributes + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/ddl_attributes + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/ddl_attributes/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/ddl_attributes/storage_test/ticdc-ddl-attributes-test-5389?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' + set +x + tso='449576008354365441 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576008354365441 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x wait process cdc.test exit for 2-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ff71a40013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-njdk7, pid:3888, start at 2024-05-06 19:52:50.565791081 +0800 CST m=+25.399926217 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:50.573 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:50.537 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:50.537 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ff71a40013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-njdk7, pid:3888, start at 2024-05-06 19:52:50.565791081 +0800 CST m=+25.399926217 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:50.573 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:50.537 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:50.537 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fe545c0002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-njdk7, pid:3953, start at 2024-05-06 19:52:32.280356495 +0800 CST m=+7.030696990 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:32.289 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:32.279 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:32.279 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_rocks/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_rocks/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/multi_rocks/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_rocks/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/multi_rocks/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 3-th time... try a VALID cdc server command [Mon May 6 19:52:51 CST 2024] <<<<<< START cdc server in cdc_server_tips case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.54285430.out server --log-file /tmp/tidb_cdc_test/cdc_server_tips/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cdc_server_tips/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 start tidb cluster in /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:52:52 CST 2024] <<<<<< run test case changefeed_error success! >>>>>> check diff failed 3-th time, retry later check diff failed 4-th time, retry later Verifying downstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Mon May 6 19:52:53 CST 2024] <<<<<< START cdc server in multi_rocks case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_rocks.1743517437.out server --log-file /tmp/tidb_cdc_test/multi_rocks/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_rocks/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff failed 5-th time, retry later check diff failed 4-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:52:54 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/672d0911-32ca-4e47-9b99-ab842309c378 {"id":"672d0911-32ca-4e47-9b99-ab842309c378","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996372} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbf9d9848 672d0911-32ca-4e47-9b99-ab842309c378 /tidb/cdc/default/default/upstream/7365853253534149748 {"id":7365853253534149748,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/672d0911-32ca-4e47-9b99-ab842309c378 {"id":"672d0911-32ca-4e47-9b99-ab842309c378","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996372} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbf9d9848 672d0911-32ca-4e47-9b99-ab842309c378 /tidb/cdc/default/default/upstream/7365853253534149748 {"id":7365853253534149748,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/672d0911-32ca-4e47-9b99-ab842309c378 {"id":"672d0911-32ca-4e47-9b99-ab842309c378","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996372} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbf9d9848 672d0911-32ca-4e47-9b99-ab842309c378 /tidb/cdc/default/default/upstream/7365853253534149748 {"id":7365853253534149748,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ffb4400017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-b8w74, pid:6689, start at 2024-05-06 19:52:54.831725733 +0800 CST m=+5.094120248 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:54.839 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:54.800 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:54.800 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ffb4400017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-b8w74, pid:6689, start at 2024-05-06 19:52:54.831725733 +0800 CST m=+5.094120248 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:54.839 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:54.800 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:54.800 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ffb80c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-b8w74, pid:6775, start at 2024-05-06 19:52:55.072341137 +0800 CST m=+5.254300176 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:55.080 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:55.043 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:55.043 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/canal_json_storage_basic/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/canal_json_storage_basic/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/canal_json_storage_basic/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/canal_json_storage_basic/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/canal_json_storage_basic/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 6-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:52:56 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4e40e9d1-ccd4-4174-90cd-1b1a71b25789 {"id":"4e40e9d1-ccd4-4174-90cd-1b1a71b25789","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996373} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbf61575f 4e40e9d1-ccd4-4174-90cd-1b1a71b25789 /tidb/cdc/default/default/upstream/7365853196823614687 {"id":7365853196823614687,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4e40e9d1-ccd4-4174-90cd-1b1a71b25789 {"id":"4e40e9d1-ccd4-4174-90cd-1b1a71b25789","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996373} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbf61575f 4e40e9d1-ccd4-4174-90cd-1b1a71b25789 /tidb/cdc/default/default/upstream/7365853196823614687 {"id":7365853196823614687,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4e40e9d1-ccd4-4174-90cd-1b1a71b25789 {"id":"4e40e9d1-ccd4-4174-90cd-1b1a71b25789","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996373} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbf61575f 4e40e9d1-ccd4-4174-90cd-1b1a71b25789 /tidb/cdc/default/default/upstream/7365853196823614687 {"id":7365853196823614687,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_rocks.cli.17722.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/multi_rocks/storage_test/ticdc-multi-rocks-test-12604?protocol=canal-json&enable-tidb-extension=true' check diff failed 5-th time, retry later Create changefeed successfully! ID: 181844b8-110a-4e97-a0ef-56e64a3a42f9 Info: {"upstream_id":7365853196823614687,"namespace":"default","id":"181844b8-110a-4e97-a0ef-56e64a3a42f9","sink_uri":"file:///tmp/tidb_cdc_test/multi_rocks/storage_test/ticdc-multi-rocks-test-12604?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:52:57.053197251+08:00","start_ts":449576010235772934,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576010235772934,"checkpoint_ts":449576010235772934,"checkpoint_time":"2024-05-06 19:52:56.937"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Mon May 6 19:52:58 CST 2024] <<<<<< START cdc server in canal_json_storage_basic case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_storage_basic.80278029.out server --log-file /tmp/tidb_cdc_test/canal_json_storage_basic/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/canal_json_storage_basic/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x + workdir=/tmp/tidb_cdc_test/multi_rocks + sink_uri='file:///tmp/tidb_cdc_test/multi_rocks/storage_test/ticdc-multi-rocks-test-12604?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/multi_rocks ++ date + echo '[Mon May 6 19:52:58 CST 2024] <<<<<< START storage consumer in multi_rocks case >>>>>>' [Mon May 6 19:52:58 CST 2024] <<<<<< START storage consumer in multi_rocks case >>>>>> + cd /tmp/tidb_cdc_test/multi_rocks + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/multi_rocks + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/multi_rocks/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/multi_rocks/storage_test/ticdc-multi-rocks-test-12604?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' check diff failed 6-th time, retry later ***************** properties ***************** "readproportion"="0" "mysql.port"="4000" "readallfields"="true" "updateproportion"="0" "insertproportion"="0" "workload"="core" "scanproportion"="0" "mysql.host"="127.0.0.1" "mysql.db"="multi_rocks" "dotransactions"="false" "recordcount"="1000" "requestdistribution"="uniform" "threadcount"="2" "mysql.user"="root" "table"="a1" "operationcount"="0" ********************************************** check diff failed 7-th time, retry later Run finished, takes 603.146036ms INSERT - Takes(s): 0.6, Count: 1000, OPS: 1695.0, Avg(us): 1173, Min(us): 841, Max(us): 13170, 95th(us): 2000, 99th(us): 2000 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo/run.sh using Sink-Type: storage... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo + source /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/synced_status_with_redo + CDC_BINARY=cdc.test + SINK_TYPE=storage + CDC_COUNT=3 + DB_COUNT=4 + trap stop_tidb_cluster EXIT + run_normal_case_and_unavailable_pd conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status_with_redo + mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo The 1 times to try to start tidb cluster... ***************** properties ***************** "insertproportion"="0" "updateproportion"="0" "workload"="core" "operationcount"="0" "mysql.port"="4000" "scanproportion"="0" "mysql.host"="127.0.0.1" "mysql.db"="multi_rocks" "requestdistribution"="uniform" "mysql.user"="root" "dotransactions"="false" "threadcount"="2" "readallfields"="true" "table"="a2" "readproportion"="0" "recordcount"="1000" ********************************************** Run finished, takes 596.550017ms INSERT - Takes(s): 0.6, Count: 1000, OPS: 1727.3, Avg(us): 1162, Min(us): 800, Max(us): 17525, 95th(us): 2000, 99th(us): 2000 ***************** properties ***************** "readallfields"="true" "updateproportion"="0" "mysql.db"="multi_rocks" "requestdistribution"="uniform" "threadcount"="2" "mysql.port"="4000" "mysql.host"="127.0.0.1" "dotransactions"="false" "operationcount"="0" "workload"="core" "recordcount"="1000" "table"="a3" "insertproportion"="0" "mysql.user"="root" "scanproportion"="0" "readproportion"="0" ********************************************** ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Run finished, takes 582.914067ms INSERT - Takes(s): 0.6, Count: 1000, OPS: 1765.6, Avg(us): 1134, Min(us): 799, Max(us): 16380, 95th(us): 2000, 99th(us): 2000 check diff failed 8-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... ***************** properties ***************** "readproportion"="0" "requestdistribution"="uniform" "operationcount"="0" "dotransactions"="false" "mysql.port"="4000" "recordcount"="1000" "mysql.user"="root" "scanproportion"="0" "readallfields"="true" "mysql.db"="multi_rocks" "workload"="core" "mysql.host"="127.0.0.1" "table"="a4" "updateproportion"="0" "threadcount"="2" "insertproportion"="0" ********************************************** check diff failed 7-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:53:01 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/501f86cf-e70a-48ab-b23e-944ed1bb8fa4 {"id":"501f86cf-e70a-48ab-b23e-944ed1bb8fa4","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996378} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbfc17c2b 501f86cf-e70a-48ab-b23e-944ed1bb8fa4 /tidb/cdc/default/default/upstream/7365853308567521610 {"id":7365853308567521610,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/501f86cf-e70a-48ab-b23e-944ed1bb8fa4 {"id":"501f86cf-e70a-48ab-b23e-944ed1bb8fa4","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996378} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbfc17c2b 501f86cf-e70a-48ab-b23e-944ed1bb8fa4 /tidb/cdc/default/default/upstream/7365853308567521610 {"id":7365853308567521610,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/501f86cf-e70a-48ab-b23e-944ed1bb8fa4 {"id":"501f86cf-e70a-48ab-b23e-944ed1bb8fa4","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996378} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbfc17c2b 501f86cf-e70a-48ab-b23e-944ed1bb8fa4 /tidb/cdc/default/default/upstream/7365853308567521610 {"id":7365853308567521610,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_storage_basic.cli.8085.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/canal_json_storage_basic/storage_test?flush-interval=5s&enable-tidb-extension=true' --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_basic/conf/changefeed.toml Run finished, takes 589.836891ms INSERT - Takes(s): 0.6, Count: 1000, OPS: 1741.4, Avg(us): 1148, Min(us): 820, Max(us): 15492, 95th(us): 2000, 99th(us): 2000 ***************** properties ***************** "dotransactions"="false" "readallfields"="true" "insertproportion"="0" "workload"="core" "mysql.db"="multi_rocks" "mysql.user"="root" "mysql.host"="127.0.0.1" "recordcount"="1000" "operationcount"="0" "table"="a5" "updateproportion"="0" "requestdistribution"="uniform" "threadcount"="2" "mysql.port"="4000" "scanproportion"="0" "readproportion"="0" ********************************************** Create changefeed successfully! ID: ff529c29-c8be-4b23-b6a5-4c1a99ec6d87 Info: {"upstream_id":7365853308567521610,"namespace":"default","id":"ff529c29-c8be-4b23-b6a5-4c1a99ec6d87","sink_uri":"file:///tmp/tidb_cdc_test/canal_json_storage_basic/storage_test?flush-interval=5s\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:53:01.588359284+08:00","start_ts":449576011431936002,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576011431936002,"checkpoint_ts":449576011431936002,"checkpoint_time":"2024-05-06 19:53:01.500"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... Run finished, takes 705.013718ms INSERT - Takes(s): 0.6, Count: 1000, OPS: 1620.1, Avg(us): 1376, Min(us): 759, Max(us): 87644, 95th(us): 2000, 99th(us): 5000 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370022e00019 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-vs88z, pid:7728, start at 2024-05-06 19:53:01.913340263 +0800 CST m=+5.851672631 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:01.926 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:01.929 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:01.929 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully check diff failed 8-th time, retry later + set +x table multi_rocks.finish_mark not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/processor_etcd_worker_delay/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + workdir=/tmp/tidb_cdc_test/canal_json_storage_basic + sink_uri='file:///tmp/tidb_cdc_test/canal_json_storage_basic/storage_test?flush-interval=5s&enable-tidb-extension=true' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_basic/conf/changefeed.toml + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/canal_json_storage_basic ++ date + echo '[Mon May 6 19:53:03 CST 2024] <<<<<< START storage consumer in canal_json_storage_basic case >>>>>>' [Mon May 6 19:53:03 CST 2024] <<<<<< START storage consumer in canal_json_storage_basic case >>>>>> + cd /tmp/tidb_cdc_test/canal_json_storage_basic + '[' /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_basic/conf/changefeed.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/canal_json_storage_basic + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/canal_json_storage_basic/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/canal_json_storage_basic/storage_test?flush-interval=5s&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_basic/conf/changefeed.toml VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370022e00019 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-vs88z, pid:7728, start at 2024-05-06 19:53:01.913340263 +0800 CST m=+5.851672631 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:01.926 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:01.929 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:01.929 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d37002cf4000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-vs88z, pid:7810, start at 2024-05-06 19:53:02.547414821 +0800 CST m=+6.413254392 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:02.557 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:02.525 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:02.525 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 1-th time, retry later check diff failed 9-th time, retry later table multi_rocks.finish_mark not exists for 2-th check, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/processor_etcd_worker_delay Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... table multi_rocks.finish_mark not exists for 3-th check, retry later [Mon May 6 19:53:06 CST 2024] <<<<<< START cdc server in csv_storage_multi_tables_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_multi_tables_ddl.91119113.out server --log-file /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 2-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... table ddl_attributes.attributes_t1_new not exists for 1-th check, retry later check diff failed 3-th time, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:53:08 CST 2024] <<<<<< run test case processor_stop_delay success! >>>>>> Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... table ddl_attributes.attributes_t1_new not exists for 2-th check, retry later Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table multi_rocks.finish_mark not exists for 4-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:53:09 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/88ce5ad7-0a32-4f4c-a708-368fcfadd4d8 {"id":"88ce5ad7-0a32-4f4c-a708-368fcfadd4d8","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996386} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbfdfdf33 88ce5ad7-0a32-4f4c-a708-368fcfadd4d8 /tidb/cdc/default/default/upstream/7365853329913466556 {"id":7365853329913466556,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/88ce5ad7-0a32-4f4c-a708-368fcfadd4d8 {"id":"88ce5ad7-0a32-4f4c-a708-368fcfadd4d8","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996386} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbfdfdf33 88ce5ad7-0a32-4f4c-a708-368fcfadd4d8 /tidb/cdc/default/default/upstream/7365853329913466556 {"id":7365853329913466556,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/88ce5ad7-0a32-4f4c-a708-368fcfadd4d8 {"id":"88ce5ad7-0a32-4f4c-a708-368fcfadd4d8","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996386} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbfdfdf33 88ce5ad7-0a32-4f4c-a708-368fcfadd4d8 /tidb/cdc/default/default/upstream/7365853329913466556 {"id":7365853329913466556,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: test-normal Info: {"upstream_id":7365853329913466556,"namespace":"default","id":"test-normal","sink_uri":"file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-normal-11931?flush-interval=5s\u0026protocol=csv","create_time":"2024-05-06T19:53:09.827240167+08:00","start_ts":449576012737150978,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["multi_tables_ddl_test.t1","multi_tables_ddl_test.t2","multi_tables_ddl_test.t3","multi_tables_ddl_test.t4","multi_tables_ddl_test.t1_7","multi_tables_ddl_test.t2_7","multi_tables_ddl_test.finish_mark"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576012737150978,"checkpoint_ts":449576012737150978,"checkpoint_time":"2024-05-06 19:53:06.479"} Create changefeed successfully! ID: test-error-1 Info: {"upstream_id":7365853329913466556,"namespace":"default","id":"test-error-1","sink_uri":"file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-error-1-103?flush-interval=5s\u0026protocol=csv","create_time":"2024-05-06T19:53:10.025655733+08:00","start_ts":449576012737150978,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["multi_tables_ddl_test.t5","multi_tables_ddl_test.t6","multi_tables_ddl_test.t7","multi_tables_ddl_test.t8"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576012737150978,"checkpoint_ts":449576012737150978,"checkpoint_time":"2024-05-06 19:53:06.479"} =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/ddl_sequence/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... Create changefeed successfully! ID: test-error-2 Info: {"upstream_id":7365853329913466556,"namespace":"default","id":"test-error-2","sink_uri":"file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-error-2-30138?flush-interval=5s\u0026protocol=csv","create_time":"2024-05-06T19:53:10.226634481+08:00","start_ts":449576012737150978,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["multi_tables_ddl_test.t9","multi_tables_ddl_test.t10"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576012737150978,"checkpoint_ts":449576012737150978,"checkpoint_time":"2024-05-06 19:53:06.479"} + workdir=/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + sink_uri='file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-normal-11931?flush-interval=5s&protocol=csv' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/normal.toml + log_suffix=1 ++ pwd + pwd=/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl ++ date + echo '[Mon May 6 19:53:10 CST 2024] <<<<<< START storage consumer in csv_storage_multi_tables_ddl case >>>>>>' [Mon May 6 19:53:10 CST 2024] <<<<<< START storage consumer in csv_storage_multi_tables_ddl case >>>>>> + cd /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + '[' /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/normal.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/cdc_storage_consumer1.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-normal-11931?flush-interval=5s&protocol=csv' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/normal.toml + workdir=/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + sink_uri='file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-error-1-103?flush-interval=5s&protocol=csv' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/error-1.toml + log_suffix=2 ++ pwd + pwd=/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl ++ date + echo '[Mon May 6 19:53:10 CST 2024] <<<<<< START storage consumer in csv_storage_multi_tables_ddl case >>>>>>' [Mon May 6 19:53:10 CST 2024] <<<<<< START storage consumer in csv_storage_multi_tables_ddl case >>>>>> + cd /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + '[' /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/error-1.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/cdc_storage_consumer2.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-error-1-103?flush-interval=5s&protocol=csv' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/error-1.toml + workdir=/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + sink_uri='file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-error-2-30138?flush-interval=5s&protocol=csv' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/error-2.toml + log_suffix=3 ++ pwd + pwd=/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl ++ date + echo '[Mon May 6 19:53:10 CST 2024] <<<<<< START storage consumer in csv_storage_multi_tables_ddl case >>>>>>' [Mon May 6 19:53:10 CST 2024] <<<<<< START storage consumer in csv_storage_multi_tables_ddl case >>>>>> + cd /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + '[' /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/error-2.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/cdc_storage_consumer3.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-error-2-30138?flush-interval=5s&protocol=csv' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/error-2.toml ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table multi_rocks.finish_mark not exists for 5-th check, retry later check diff failed 4-th time, retry later check diff failed 1-th time, retry later table ddl_attributes.attributes_t1_new exists table ddl_attributes.finish_mark not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table multi_rocks.finish_mark not exists for 6-th check, retry later check diff failed 5-th time, retry later check diff failed 2-th time, retry later table ddl_attributes.finish_mark not exists for 2-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3700ac3c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l, pid:10310, start at 2024-05-06 19:53:10.718281311 +0800 CST m=+5.494320397 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:10.728 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:10.721 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:10.721 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3700ac3c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l, pid:10310, start at 2024-05-06 19:53:10.718281311 +0800 CST m=+5.494320397 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:10.728 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:10.721 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:10.721 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3700add40014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l, pid:10385, start at 2024-05-06 19:53:10.798762138 +0800 CST m=+5.490101233 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:10.805 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:10.773 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:10.773 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/ddl_sequence Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... table multi_rocks.finish_mark not exists for 7-th check, retry later check diff failed 6-th time, retry later table ddl_attributes.finish_mark not exists for 3-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table multi_tables_ddl_test.t55 not exists for 1-th check, retry later check diff successfully [Mon May 6 19:53:15 CST 2024] <<<<<< run test case canal_json_storage_basic success! >>>>>> + cd /tmp/tidb_cdc_test/synced_status_with_redo ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.11643.out cli tso query --pd=http://127.0.0.1:2379 Verifying downstream PD is started... table multi_rocks.finish_mark not exists for 8-th check, retry later valid ~~~ running cdc Failed to start cdc, the usage tips should be printed 1st test case cdc_server_tips success! try an INVALID cdc server command [Mon May 6 19:53:15 CST 2024] <<<<<< START cdc server in cdc_server_tips case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ true != \n\o ]] + GO_FAILPOINTS= + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.55165518.out server --log-file /tmp/tidb_cdc_test/cdc_server_tips/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cdc_server_tips/cdc_data --cluster-id default --pd None check diff failed 7-th time, retry later table multi_tables_ddl_test.t55 not exists for 2-th check, retry later table ddl_attributes.finish_mark not exists for 4-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3700e7080014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-81vjm, pid:4382, start at 2024-05-06 19:53:14.483593264 +0800 CST m=+5.400091399 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:14.490 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:14.484 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:14.484 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3700e7080014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-81vjm, pid:4382, start at 2024-05-06 19:53:14.483593264 +0800 CST m=+5.400091399 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:14.490 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:14.484 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:14.484 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3700e9700014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-81vjm, pid:4456, start at 2024-05-06 19:53:14.631408725 +0800 CST m=+5.486102083 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:14.641 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:14.638 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:14.638 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + set +x + tso='449576015225421826 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + awk -F ' ' '{print $1}' + echo 449576015225421826 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + set +x + start_ts=449576015225421826 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test [Mon May 6 19:53:17 CST 2024] <<<<<< START cdc server in synced_status_with_redo case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.1168011682.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table multi_tables_ddl_test.t55 not exists for 3-th check, retry later table ddl_attributes.finish_mark not exists for 5-th check, retry later table multi_rocks.finish_mark not exists for 9-th check, retry later check diff failed 8-th time, retry later [Mon May 6 19:53:19 CST 2024] <<<<<< START cdc server in processor_etcd_worker_delay case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/orchestrator/ProcessorEtcdDelay=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_etcd_worker_delay.58275829.out server --log-file /tmp/tidb_cdc_test/processor_etcd_worker_delay/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_etcd_worker_delay/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:53:20 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3de9794a-c2e5-40f1-8d1b-31a8937113c8 {"id":"3de9794a-c2e5-40f1-8d1b-31a8937113c8","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996397} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc003c83a 3de9794a-c2e5-40f1-8d1b-31a8937113c8 /tidb/cdc/default/default/upstream/7365853376735490353 {"id":7365853376735490353,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3de9794a-c2e5-40f1-8d1b-31a8937113c8 {"id":"3de9794a-c2e5-40f1-8d1b-31a8937113c8","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996397} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc003c83a 3de9794a-c2e5-40f1-8d1b-31a8937113c8 /tidb/cdc/default/default/upstream/7365853376735490353 {"id":7365853376735490353,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3de9794a-c2e5-40f1-8d1b-31a8937113c8 {"id":"3de9794a-c2e5-40f1-8d1b-31a8937113c8","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996397} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc003c83a 3de9794a-c2e5-40f1-8d1b-31a8937113c8 /tidb/cdc/default/default/upstream/7365853376735490353 {"id":7365853376735490353,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449576015225421826 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.11738.out cli changefeed create --start-ts=449576015225421826 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml table multi_rocks.finish_mark not exists for 10-th check, retry later table ddl_attributes.finish_mark not exists for 6-th check, retry later Create changefeed successfully! ID: test-1 Info: {"upstream_id":7365853376735490353,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-06T19:53:21.13938397+08:00","start_ts":449576015225421826,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576015225421826,"checkpoint_ts":449576015225421826,"checkpoint_time":"2024-05-06 19:53:15.971"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... check diff failed 9-th time, retry later table multi_tables_ddl_test.t55 not exists for 4-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > + set +x ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:53:22 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/85ffcfa1-b672-40e9-85fa-ae9025c7ba64 {"id":"85ffcfa1-b672-40e9-85fa-ae9025c7ba64","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996399} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc016c82e 85ffcfa1-b672-40e9-85fa-ae9025c7ba64 /tidb/cdc/default/default/upstream/7365853389110001837 {"id":7365853389110001837,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/85ffcfa1-b672-40e9-85fa-ae9025c7ba64 {"id":"85ffcfa1-b672-40e9-85fa-ae9025c7ba64","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996399} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc016c82e 85ffcfa1-b672-40e9-85fa-ae9025c7ba64 /tidb/cdc/default/default/upstream/7365853389110001837 {"id":7365853389110001837,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/85ffcfa1-b672-40e9-85fa-ae9025c7ba64 {"id":"85ffcfa1-b672-40e9-85fa-ae9025c7ba64","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996399} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc016c82e 85ffcfa1-b672-40e9-85fa-ae9025c7ba64 /tidb/cdc/default/default/upstream/7365853389110001837 {"id":7365853389110001837,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + workdir=/tmp/tidb_cdc_test/processor_etcd_worker_delay + sink_uri='file:///tmp/tidb_cdc_test/processor_etcd_worker_delay/storage_test/processor-etcd-delay-11019?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/processor_etcd_worker_delay ++ date + echo '[Mon May 6 19:53:22 CST 2024] <<<<<< START storage consumer in processor_etcd_worker_delay case >>>>>>' [Mon May 6 19:53:22 CST 2024] <<<<<< START storage consumer in processor_etcd_worker_delay case >>>>>> + cd /tmp/tidb_cdc_test/processor_etcd_worker_delay + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/processor_etcd_worker_delay + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/processor_etcd_worker_delay/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/processor_etcd_worker_delay/storage_test/processor-etcd-delay-11019?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 1554 0 --:--:-- --:--:-- --:--:-- 1556 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-06 19:53:22.272","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-05-06 19:53:22.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '19:53:22.272","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-06' '19:53:22.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '19:53:22.272","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-06' '19:53:22.000","info":"Data' syncing is 'finished"}' ++ jq -r .sink_checkpoint_ts + sink_checkpoint_ts='2024-05-06 19:53:22.272' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '19:53:22.272","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-06' '19:53:22.000","info":"Data' syncing is 'finished"}' ++ jq -r .puller_resolved_ts + puller_resolved_ts='1970-01-01 08:00:00.000' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '19:53:22.272","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-06' '19:53:22.000","info":"Data' syncing is 'finished"}' ++ jq -r .last_synced_ts + last_synced_ts='1970-01-01 08:00:00.000' + '[' true '!=' true ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' ++ date '+%Y-%m-%d %H:%M:%S' + current='2024-05-06 19:53:22' + echo 'sink_checkpoint_ts is 2024-05-06' 19:53:22.272 sink_checkpoint_ts is 2024-05-06 19:53:22.272 ++ date -d '2024-05-06 19:53:22.272' +%s + checkpoint_timestamp=1714996402 ++ date -d '2024-05-06 19:53:22' +%s + current_timestamp=1714996402 + '[' 0 -gt 300 ']' + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' table multi_rocks.finish_mark not exists for 11-th check, retry later check diff successfully table multi_tables_ddl_test.t55 not exists for 5-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/partition_table/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table ddl_attributes.finish_mark not exists for 7-th check, retry later + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table multi_rocks.finish_mark not exists for 12-th check, retry later check diff failed 1-th time, retry later table multi_tables_ddl_test.t55 not exists for 6-th check, retry later table ddl_attributes.finish_mark not exists for 8-th check, retry later table test.t1 exists + sleep 5 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/partition_table Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... table multi_tables_ddl_test.t55 not exists for 7-th check, retry later table multi_rocks.finish_mark not exists for 13-th check, retry later table ddl_attributes.finish_mark not exists for 9-th check, retry later check diff failed 2-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d37018f98000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-drnnn, pid:7730, start at 2024-05-06 19:53:25.237048016 +0800 CST m=+5.342948532 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:25.244 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:25.222 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:25.222 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d37018f98000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-drnnn, pid:7730, start at 2024-05-06 19:53:25.237048016 +0800 CST m=+5.342948532 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:25.244 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:25.222 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:25.222 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370190440014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-drnnn, pid:7810, start at 2024-05-06 19:53:25.293515807 +0800 CST m=+5.344165195 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:25.301 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:25.265 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:25.265 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_sequence/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_sequence/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table multi_tables_ddl_test.t55 not exists for 8-th check, retry later table multi_rocks.finish_mark not exists for 14-th check, retry later table ddl_attributes.finish_mark not exists for 10-th check, retry later check diff failed 3-th time, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.cli.9196.out cli tso query --pd=http://127.0.0.1:2379 Starting Upstream TiDB... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_partition_table/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table multi_tables_ddl_test.t55 exists table multi_tables_ddl_test.t66 exists table multi_tables_ddl_test.t7 exists table multi_tables_ddl_test.t88 not exists for 1-th check, retry later Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table multi_rocks.finish_mark exists check diff successfully table ddl_attributes.finish_mark exists check diff failed 4-th time, retry later ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 2420 0 --:--:-- --:--:-- --:--:-- 2430 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-06 19:53:28.322","puller_resolved_ts":"2024-05-06 19:53:22.921","last_synced_ts":"2024-05-06 19:53:23.371","now_ts":"2024-05-06 19:53:30.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '19:53:28.322","puller_resolved_ts":"2024-05-06' '19:53:22.921","last_synced_ts":"2024-05-06' '19:53:23.371","now_ts":"2024-05-06' '19:53:30.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '19:53:28.322","puller_resolved_ts":"2024-05-06' '19:53:22.921","last_synced_ts":"2024-05-06' '19:53:23.371","now_ts":"2024-05-06' '19:53:30.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... + set +x + tso='449576019052724225 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576019052724225 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:53:32 CST 2024] <<<<<< START cdc server in ddl_sequence case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.92289230.out server --log-file /tmp/tidb_cdc_test/ddl_sequence/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_sequence/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 1-th time... wait process cdc.test exit for 3-th time... wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:53:33 CST 2024] <<<<<< run test case multi_rocks success! >>>>>> check_changefeed_state http://127.0.0.1:2379 0c7e4d01-1056-4f05-a8ac-22fcfe4649da finished null + endpoints=http://127.0.0.1:2379 + changefeed_id=0c7e4d01-1056-4f05-a8ac-22fcfe4649da + expected_state=finished + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 0c7e4d01-1056-4f05-a8ac-22fcfe4649da -s + info='{ "upstream_id": 7365852967548068780, "namespace": "default", "id": "0c7e4d01-1056-4f05-a8ac-22fcfe4649da", "state": "finished", "checkpoint_tso": 449576014839545861, "checkpoint_time": "2024-05-06 19:53:14.499", "error": null }' + echo '{ "upstream_id": 7365852967548068780, "namespace": "default", "id": "0c7e4d01-1056-4f05-a8ac-22fcfe4649da", "state": "finished", "checkpoint_tso": 449576014839545861, "checkpoint_time": "2024-05-06 19:53:14.499", "error": null }' { "upstream_id": 7365852967548068780, "namespace": "default", "id": "0c7e4d01-1056-4f05-a8ac-22fcfe4649da", "state": "finished", "checkpoint_tso": 449576014839545861, "checkpoint_time": "2024-05-06 19:53:14.499", "error": null } ++ echo '{' '"upstream_id":' 7365852967548068780, '"namespace":' '"default",' '"id":' '"0c7e4d01-1056-4f05-a8ac-22fcfe4649da",' '"state":' '"finished",' '"checkpoint_tso":' 449576014839545861, '"checkpoint_time":' '"2024-05-06' '19:53:14.499",' '"error":' null '}' ++ jq -r .state + state=finished + [[ ! finished == \f\i\n\i\s\h\e\d ]] ++ echo '{' '"upstream_id":' 7365852967548068780, '"namespace":' '"default",' '"id":' '"0c7e4d01-1056-4f05-a8ac-22fcfe4649da",' '"state":' '"finished",' '"checkpoint_tso":' 449576014839545861, '"checkpoint_time":' '"2024-05-06' '19:53:14.499",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:53:28 CST 2024] <<<<<< run test case changefeed_finish success! >>>>>> table multi_tables_ddl_test.t88 exists table multi_tables_ddl_test.finish_mark not exists for 1-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:53:33 CST 2024] <<<<<< run test case ddl_attributes success! >>>>>> check diff failed 5-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/canal_json_storage_partition_table Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... table multi_tables_ddl_test.finish_mark not exists for 2-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:53:35 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d092f875-ed6f-4332-a6cc-637e9cf6e48c {"id":"d092f875-ed6f-4332-a6cc-637e9cf6e48c","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996412} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc037263d d092f875-ed6f-4332-a6cc-637e9cf6e48c /tidb/cdc/default/default/upstream/7365853433111351516 {"id":7365853433111351516,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d092f875-ed6f-4332-a6cc-637e9cf6e48c {"id":"d092f875-ed6f-4332-a6cc-637e9cf6e48c","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996412} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc037263d d092f875-ed6f-4332-a6cc-637e9cf6e48c /tidb/cdc/default/default/upstream/7365853433111351516 {"id":7365853433111351516,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d092f875-ed6f-4332-a6cc-637e9cf6e48c {"id":"d092f875-ed6f-4332-a6cc-637e9cf6e48c","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996412} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc037263d d092f875-ed6f-4332-a6cc-637e9cf6e48c /tidb/cdc/default/default/upstream/7365853433111351516 {"id":7365853433111351516,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.cli.9288.out cli changefeed create --start-ts=449576019052724225 '--sink-uri=file:///tmp/tidb_cdc_test/ddl_sequence/storage_test/ticdc-ddl-sequence-test-7514?protocol=canal-json&enable-tidb-extension=true' ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Create changefeed successfully! ID: 04481a62-4dc8-4051-988f-ea7b2d643bea Info: {"upstream_id":7365853433111351516,"namespace":"default","id":"04481a62-4dc8-4051-988f-ea7b2d643bea","sink_uri":"file:///tmp/tidb_cdc_test/ddl_sequence/storage_test/ticdc-ddl-sequence-test-7514?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:53:35.552818276+08:00","start_ts":449576019052724225,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576019052724225,"checkpoint_ts":449576019052724225,"checkpoint_time":"2024-05-06 19:53:30.571"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... check diff failed 6-th time, retry later + set +x + workdir=/tmp/tidb_cdc_test/ddl_sequence + sink_uri='file:///tmp/tidb_cdc_test/ddl_sequence/storage_test/ticdc-ddl-sequence-test-7514?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/ddl_sequence ++ date + echo '[Mon May 6 19:53:36 CST 2024] <<<<<< START storage consumer in ddl_sequence case >>>>>>' [Mon May 6 19:53:36 CST 2024] <<<<<< START storage consumer in ddl_sequence case >>>>>> + cd /tmp/tidb_cdc_test/ddl_sequence + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/ddl_sequence + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/ddl_sequence/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/ddl_sequence/storage_test/ticdc-ddl-sequence-test-7514?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table multi_tables_ddl_test.finish_mark not exists for 3-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d37023c8c0015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-t7zm9, pid:9527, start at 2024-05-06 19:53:36.319013099 +0800 CST m=+5.483428286 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:36.328 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:36.291 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:36.291 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 7-th time, retry later invalid ~~~ running cdc Failed to start cdc, the usage tips should be printed 2nd test case cdc_server_tips success! [Mon May 6 19:53:35 CST 2024] <<<<<< run all test cases cdc_server_tips success! >>>>>> table multi_tables_ddl_test.finish_mark not exists for 4-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d37023c8c0015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-t7zm9, pid:9527, start at 2024-05-06 19:53:36.319013099 +0800 CST m=+5.483428286 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:36.328 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:36.291 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:36.291 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d37023dfc0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-t7zm9, pid:9606, start at 2024-05-06 19:53:36.411829921 +0800 CST m=+5.498014712 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:36.419 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:36.383 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:36.383 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/partition_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/partition_table/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff failed 8-th time, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/force_replicate_table/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table multi_tables_ddl_test.finish_mark not exists for 5-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.cli.10920.out cli tso query --pd=http://127.0.0.1:2379 check diff failed 9-th time, retry later + set +x + tso='449576021928443906 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576021928443906 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:53:43 CST 2024] <<<<<< START cdc server in partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.1097610978.out server --log-file /tmp/tidb_cdc_test/partition_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/partition_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table multi_tables_ddl_test.finish_mark exists check table exists success + endpoints=http://127.0.0.1:2379 + changefeed_id=test-normal + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c test-normal -s + info='{ "upstream_id": 7365853329913466556, "namespace": "default", "id": "test-normal", "state": "normal", "checkpoint_tso": 449576022095691780, "checkpoint_time": "2024-05-06 19:53:42.179", "error": null }' + echo '{ "upstream_id": 7365853329913466556, "namespace": "default", "id": "test-normal", "state": "normal", "checkpoint_tso": 449576022095691780, "checkpoint_time": "2024-05-06 19:53:42.179", "error": null }' { "upstream_id": 7365853329913466556, "namespace": "default", "id": "test-normal", "state": "normal", "checkpoint_tso": 449576022095691780, "checkpoint_time": "2024-05-06 19:53:42.179", "error": null } ++ echo '{' '"upstream_id":' 7365853329913466556, '"namespace":' '"default",' '"id":' '"test-normal",' '"state":' '"normal",' '"checkpoint_tso":' 449576022095691780, '"checkpoint_time":' '"2024-05-06' '19:53:42.179",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7365853329913466556, '"namespace":' '"default",' '"id":' '"test-normal",' '"state":' '"normal",' '"checkpoint_tso":' 449576022095691780, '"checkpoint_time":' '"2024-05-06' '19:53:42.179",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + endpoints=http://127.0.0.1:2379 + changefeed_id=test-error-1 + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c test-error-1 -s table ddl_sequence.finish_mark not exists for 1-th check, retry later + info='{ "upstream_id": 7365853329913466556, "namespace": "default", "id": "test-error-1", "state": "normal", "checkpoint_tso": 449576022095691782, "checkpoint_time": "2024-05-06 19:53:42.179", "error": null }' + echo '{ "upstream_id": 7365853329913466556, "namespace": "default", "id": "test-error-1", "state": "normal", "checkpoint_tso": 449576022095691782, "checkpoint_time": "2024-05-06 19:53:42.179", "error": null }' { "upstream_id": 7365853329913466556, "namespace": "default", "id": "test-error-1", "state": "normal", "checkpoint_tso": 449576022095691782, "checkpoint_time": "2024-05-06 19:53:42.179", "error": null } ++ echo '{' '"upstream_id":' 7365853329913466556, '"namespace":' '"default",' '"id":' '"test-error-1",' '"state":' '"normal",' '"checkpoint_tso":' 449576022095691782, '"checkpoint_time":' '"2024-05-06' '19:53:42.179",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7365853329913466556, '"namespace":' '"default",' '"id":' '"test-error-1",' '"state":' '"normal",' '"checkpoint_tso":' 449576022095691782, '"checkpoint_time":' '"2024-05-06' '19:53:42.179",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + endpoints=http://127.0.0.1:2379 + changefeed_id=test-error-2 + expected_state=failed + error_msg=ErrSyncRenameTableFailed + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c test-error-2 -s ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/force_replicate_table Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... + info='{ "upstream_id": 7365853329913466556, "namespace": "default", "id": "test-error-2", "state": "failed", "checkpoint_tso": 449576014742552604, "checkpoint_time": "2024-05-06 19:53:14.129", "error": { "time": "2024-05-06T19:53:15.199233681+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSyncRenameTableFailed", "message": "[CDC:ErrSyncRenameTableFailed]table'\''s old name is not in filter rule, and its new name in filter rule table id '\''130'\'', ddl query: [rename table t11 to t9], it'\''s an unexpected behavior, if you want to replicate this table, please add its old name to filter rule." } }' + echo '{ "upstream_id": 7365853329913466556, "namespace": "default", "id": "test-error-2", "state": "failed", "checkpoint_tso": 449576014742552604, "checkpoint_time": "2024-05-06 19:53:14.129", "error": { "time": "2024-05-06T19:53:15.199233681+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSyncRenameTableFailed", "message": "[CDC:ErrSyncRenameTableFailed]table'\''s old name is not in filter rule, and its new name in filter rule table id '\''130'\'', ddl query: [rename table t11 to t9], it'\''s an unexpected behavior, if you want to replicate this table, please add its old name to filter rule." } }' { "upstream_id": 7365853329913466556, "namespace": "default", "id": "test-error-2", "state": "failed", "checkpoint_tso": 449576014742552604, "checkpoint_time": "2024-05-06 19:53:14.129", "error": { "time": "2024-05-06T19:53:15.199233681+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSyncRenameTableFailed", "message": "[CDC:ErrSyncRenameTableFailed]table's old name is not in filter rule, and its new name in filter rule table id '130', ddl query: [rename table t11 to t9], it's an unexpected behavior, if you want to replicate this table, please add its old name to filter rule." } } ++ jq -r .state ++ echo '{' '"upstream_id":' 7365853329913466556, '"namespace":' '"default",' '"id":' '"test-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 449576014742552604, '"checkpoint_time":' '"2024-05-06' '19:53:14.129",' '"error":' '{' '"time":' '"2024-05-06T19:53:15.199233681+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSyncRenameTableFailed",' '"message":' '"[CDC:ErrSyncRenameTableFailed]table'\''s' old name is not in filter rule, and its new name in filter rule table id ''\''130'\'',' ddl query: '[rename' table t11 to 't9],' 'it'\''s' an unexpected behavior, if you want to replicate this table, please add its old name to filter 'rule."' '}' '}' + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ jq -r .error.message ++ echo '{' '"upstream_id":' 7365853329913466556, '"namespace":' '"default",' '"id":' '"test-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 449576014742552604, '"checkpoint_time":' '"2024-05-06' '19:53:14.129",' '"error":' '{' '"time":' '"2024-05-06T19:53:15.199233681+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSyncRenameTableFailed",' '"message":' '"[CDC:ErrSyncRenameTableFailed]table'\''s' old name is not in filter rule, and its new name in filter rule table id ''\''130'\'',' ddl query: '[rename' table t11 to 't9],' 'it'\''s' an unexpected behavior, if you want to replicate this table, please add its old name to filter 'rule."' '}' '}' + message='[CDC:ErrSyncRenameTableFailed]table'\''s old name is not in filter rule, and its new name in filter rule table id '\''130'\'', ddl query: [rename table t11 to t9], it'\''s an unexpected behavior, if you want to replicate this table, please add its old name to filter rule.' + [[ ! [CDC:ErrSyncRenameTableFailed]table's old name is not in filter rule, and its new name in filter rule table id '130', ddl query: [rename table t11 to t9], it's an unexpected behavior, if you want to replicate this table, please add its old name to filter rule. =~ ErrSyncRenameTableFailed ]] check diff successfully check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... \033[0;36m<<< Run all test success >>>\033[0m table ddl_sequence.finish_mark not exists for 2-th check, retry later [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } wait process cdc.test exit for 3-th time... [Pipeline] // dir [Pipeline] } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } check diff failed 1-th time, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] // stage [Pipeline] } [Pipeline] // container cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:53:46 CST 2024] <<<<<< run test case csv_storage_multi_tables_ddl success! >>>>>> [Pipeline] } [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc already exists) [Pipeline] // withEnv [Pipeline] // cache [Pipeline] } [Pipeline] } [Pipeline] // node [Pipeline] // dir + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:53:46 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9ace3ca5-2eb4-41cc-8383-10526d9ed445 {"id":"9ace3ca5-2eb4-41cc-8383-10526d9ed445","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996423} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc069d03b 9ace3ca5-2eb4-41cc-8383-10526d9ed445 /tidb/cdc/default/default/upstream/7365853483331440383 {"id":7365853483331440383,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9ace3ca5-2eb4-41cc-8383-10526d9ed445 {"id":"9ace3ca5-2eb4-41cc-8383-10526d9ed445","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996423} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc069d03b 9ace3ca5-2eb4-41cc-8383-10526d9ed445 /tidb/cdc/default/default/upstream/7365853483331440383 {"id":7365853483331440383,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9ace3ca5-2eb4-41cc-8383-10526d9ed445 {"id":"9ace3ca5-2eb4-41cc-8383-10526d9ed445","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996423} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc069d03b 9ace3ca5-2eb4-41cc-8383-10526d9ed445 /tidb/cdc/default/default/upstream/7365853483331440383 {"id":7365853483331440383,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.cli.11030.out cli changefeed create --start-ts=449576021928443906 '--sink-uri=file:///tmp/tidb_cdc_test/partition_table/storage_test/ticdc-partition-table-test-25824?protocol=canal-json&enable-tidb-extension=true' [Pipeline] } [Pipeline] } [Pipeline] // podTemplate [Pipeline] // withCredentials [Pipeline] } [Pipeline] } Create changefeed successfully! ID: e8741b11-2eda-47ba-8c24-aa9ef5864fde Info: {"upstream_id":7365853483331440383,"namespace":"default","id":"e8741b11-2eda-47ba-8c24-aa9ef5864fde","sink_uri":"file:///tmp/tidb_cdc_test/partition_table/storage_test/ticdc-partition-table-test-25824?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:53:46.615390797+08:00","start_ts":449576021928443906,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576021928443906,"checkpoint_ts":449576021928443906,"checkpoint_time":"2024-05-06 19:53:41.541"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... [Pipeline] // withEnv [Pipeline] // timeout [Pipeline] } [Pipeline] } [Pipeline] // stage [Pipeline] // stage [Pipeline] } [Pipeline] } Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } table ddl_sequence.finish_mark not exists for 3-th check, retry later [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } + set +x + workdir=/tmp/tidb_cdc_test/partition_table + sink_uri='file:///tmp/tidb_cdc_test/partition_table/storage_test/ticdc-partition-table-test-25824?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/partition_table ++ date + echo '[Mon May 6 19:53:48 CST 2024] <<<<<< START storage consumer in partition_table case >>>>>>' [Mon May 6 19:53:48 CST 2024] <<<<<< START storage consumer in partition_table case >>>>>> + cd /tmp/tidb_cdc_test/partition_table + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/partition_table + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/partition_table/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/partition_table/storage_test/ticdc-partition-table-test-25824?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/event_filter/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table ddl_sequence.finish_mark not exists for 4-th check, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3702dc940014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-b8w74, pid:9165, start at 2024-05-06 19:53:46.566020818 +0800 CST m=+5.382847152 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:46.573 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:46.533 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:46.533 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3702dc940014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-b8w74, pid:9165, start at 2024-05-06 19:53:46.566020818 +0800 CST m=+5.382847152 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:46.573 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:46.533 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:46.533 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3702ded4000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-b8w74, pid:9229, start at 2024-05-06 19:53:46.687084345 +0800 CST m=+5.447448519 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:46.694 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:46.677 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:46.677 +0800 All versions after safe point can be accessed. (DO NOT EDIT) check diff failed 3-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/event_filter Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/canal_json_storage_partition_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/canal_json_storage_partition_table/tiflash/log/error.log table ddl_sequence.finish_mark not exists for 5-th check, retry later arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/canal_json_storage_partition_table/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/canal_json_storage_partition_table/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/canal_json_storage_partition_table/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 4-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_sequence.finish_mark not exists for 6-th check, retry later check diff failed 5-th time, retry later table processor_delay.t1 exists table processor_delay.t2 exists table processor_delay.t3 exists table processor_delay.t4 not exists for 1-th check, retry later table processor_delay.t4 exists table processor_delay.t5 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370356500012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-7rfgh, pid:4161, start at 2024-05-06 19:53:54.354961093 +0800 CST m=+5.239630672 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:54.362 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:54.324 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:54.324 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t5 exists table processor_delay.t6 not exists for 1-th check, retry later table ddl_sequence.finish_mark not exists for 7-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff failed 6-th time, retry later table ddl_sequence.finish_mark not exists for 8-th check, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370356500012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-7rfgh, pid:4161, start at 2024-05-06 19:53:54.354961093 +0800 CST m=+5.239630672 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:54.362 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. table processor_delay.t6 not exists for 2-th check, retry later tikv_gc_last_run_time 20240506-19:53:54.324 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:54.324 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370358900014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-7rfgh, pid:4254, start at 2024-05-06 19:53:54.490129586 +0800 CST m=+5.318451252 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:54.496 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:54.468 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:54.468 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/force_replicate_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/force_replicate_table/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table partition_table.t not exists for 1-th check, retry later [Mon May 6 19:53:57 CST 2024] <<<<<< START cdc server in canal_json_storage_partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_storage_partition_table.1059310595.out server --log-file /tmp/tidb_cdc_test/canal_json_storage_partition_table/cdccdc0.log --log-level debug --data-dir /tmp/tidb_cdc_test/canal_json_storage_partition_table/cdc_datacdc0 --cluster-id default --addr 127.0.0.1:8300 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 7-th time, retry later table partition_table.t exists table partition_table.t1 not exists for 1-th check, retry later table ddl_sequence.finish_mark not exists for 9-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t6 not exists for 3-th check, retry later [Mon May 6 19:53:59 CST 2024] <<<<<< START cdc server in force_replicate_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.force_replicate_table.55365538.out server --log-file /tmp/tidb_cdc_test/force_replicate_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/force_replicate_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > check diff failed 8-th time, retry later table partition_table.t1 exists table partition_table.t2 not exists for 1-th check, retry later < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:54:00 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f72793a5-7503-4aca-8639-086db4a7bac6 {"id":"f72793a5-7503-4aca-8639-086db4a7bac6","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996439} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc08a6851 f72793a5-7503-4aca-8639-086db4a7bac6 /tidb/cdc/default/default/upstream/7365853528650801719 {"id":7365853528650801719,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f72793a5-7503-4aca-8639-086db4a7bac6 {"id":"f72793a5-7503-4aca-8639-086db4a7bac6","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996439} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc08a6851 f72793a5-7503-4aca-8639-086db4a7bac6 /tidb/cdc/default/default/upstream/7365853528650801719 {"id":7365853528650801719,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f72793a5-7503-4aca-8639-086db4a7bac6 {"id":"f72793a5-7503-4aca-8639-086db4a7bac6","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996439} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc08a6851 f72793a5-7503-4aca-8639-086db4a7bac6 /tidb/cdc/default/default/upstream/7365853528650801719 {"id":7365853528650801719,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Mon May 6 19:54:00 CST 2024] <<<<<< START cdc server in canal_json_storage_partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_storage_partition_table.1063010632.out server --log-file /tmp/tidb_cdc_test/canal_json_storage_partition_table/cdccdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/canal_json_storage_partition_table/cdc_datacdc1 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t6 not exists for 4-th check, retry later table ddl_sequence.finish_mark not exists for 10-th check, retry later check diff failed 9-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:54:02 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/db1db968-8c78-4a42-9947-d6ece9c8facf {"id":"db1db968-8c78-4a42-9947-d6ece9c8facf","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996439} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc0ab6034 db1db968-8c78-4a42-9947-d6ece9c8facf /tidb/cdc/default/default/upstream/7365853565836689787 {"id":7365853565836689787,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/db1db968-8c78-4a42-9947-d6ece9c8facf {"id":"db1db968-8c78-4a42-9947-d6ece9c8facf","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996439} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc0ab6034 db1db968-8c78-4a42-9947-d6ece9c8facf /tidb/cdc/default/default/upstream/7365853565836689787 {"id":7365853565836689787,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/db1db968-8c78-4a42-9947-d6ece9c8facf {"id":"db1db968-8c78-4a42-9947-d6ece9c8facf","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996439} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc0ab6034 db1db968-8c78-4a42-9947-d6ece9c8facf /tidb/cdc/default/default/upstream/7365853565836689787 {"id":7365853565836689787,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: 1530672f-84b4-421d-856c-f2b175363224 Info: {"upstream_id":7365853565836689787,"namespace":"default","id":"1530672f-84b4-421d-856c-f2b175363224","sink_uri":"file:///tmp/tidb_cdc_test/force_replicate_table/storage_test/ticdc-force_replicate_table-test-27150?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:54:02.789569355+08:00","start_ts":449576026629472258,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":true,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576026629472258,"checkpoint_ts":449576026629472258,"checkpoint_time":"2024-05-06 19:53:59.474"} + workdir=/tmp/tidb_cdc_test/force_replicate_table + sink_uri='file:///tmp/tidb_cdc_test/force_replicate_table/storage_test/ticdc-force_replicate_table-test-27150?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/force_replicate_table/conf/changefeed.toml + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/force_replicate_table ++ date + echo '[Mon May 6 19:54:02 CST 2024] <<<<<< START storage consumer in force_replicate_table case >>>>>>' [Mon May 6 19:54:02 CST 2024] <<<<<< START storage consumer in force_replicate_table case >>>>>> + cd /tmp/tidb_cdc_test/force_replicate_table + '[' /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/force_replicate_table/conf/changefeed.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/force_replicate_table + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/force_replicate_table/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/force_replicate_table/storage_test/ticdc-force_replicate_table-test-27150?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/force_replicate_table/conf/changefeed.toml ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.t2 not exists for 2-th check, retry later table processor_delay.t6 exists table processor_delay.t7 exists table processor_delay.t8 not exists for 1-th check, retry later table ddl_sequence.finish_mark not exists for 11-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:54:03 GMT < Content-Length: 1299 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7a310c13-b8e5-4cb2-9d98-54c8055ead45 {"id":"7a310c13-b8e5-4cb2-9d98-54c8055ead45","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996441} /tidb/cdc/default/__cdc_meta__/capture/f72793a5-7503-4aca-8639-086db4a7bac6 {"id":"f72793a5-7503-4aca-8639-086db4a7bac6","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996439} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc08a6851 f72793a5-7503-4aca-8639-086db4a7bac6 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc08a687e 7a310c13-b8e5-4cb2-9d98-54c8055ead45 /tidb/cdc/default/default/upstream/7365853528650801719 {"id":7365853528650801719,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7a310c13-b8e5-4cb2-9d98-54c8055ead45 {"id":"7a310c13-b8e5-4cb2-9d98-54c8055ead45","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996441} /tidb/cdc/default/__cdc_meta__/capture/f72793a5-7503-4aca-8639-086db4a7bac6 {"id":"f72793a5-7503-4aca-8639-086db4a7bac6","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996439} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc08a6851 f72793a5-7503-4aca-8639-086db4a7bac6 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc08a687e 7a310c13-b8e5-4cb2-9d98-54c8055ead45 /tidb/cdc/default/default/upstream/7365853528650801719 {"id":7365853528650801719,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7a310c13-b8e5-4cb2-9d98-54c8055ead45 {"id":"7a310c13-b8e5-4cb2-9d98-54c8055ead45","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996441} /tidb/cdc/default/__cdc_meta__/capture/f72793a5-7503-4aca-8639-086db4a7bac6 {"id":"f72793a5-7503-4aca-8639-086db4a7bac6","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996439} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc08a6851 f72793a5-7503-4aca-8639-086db4a7bac6 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc08a687e 7a310c13-b8e5-4cb2-9d98-54c8055ead45 /tidb/cdc/default/default/upstream/7365853528650801719 {"id":7365853528650801719,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Mon May 6 19:54:03 CST 2024] <<<<<< START cdc server in canal_json_storage_partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_storage_partition_table.1069010692.out server --log-file /tmp/tidb_cdc_test/canal_json_storage_partition_table/cdccdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/canal_json_storage_partition_table/cdc_datacdc2 --cluster-id default --addr 127.0.0.1:8302 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_partition_table/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... check diff successfully table force_replicate_table.t0 not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/csv_storage_partition_table Starting Upstream PD... table partition_table.t2 not exists for 3-th check, retry later table processor_delay.t8 not exists for 2-th check, retry later Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... table ddl_sequence.finish_mark not exists for 12-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3703d2140005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3, pid:6432, start at 2024-05-06 19:54:02.25151228 +0800 CST m=+5.311191438 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:56:02.258 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:54:02.245 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:44:02.245 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3703d2140005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3, pid:6432, start at 2024-05-06 19:54:02.25151228 +0800 CST m=+5.311191438 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:56:02.258 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:54:02.245 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:44:02.245 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3703d1380018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3, pid:6513, start at 2024-05-06 19:54:02.235938517 +0800 CST m=+5.247099501 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:56:02.246 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:54:02.240 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:44:02.240 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/event_filter/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/event_filter/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 1-th time, retry later table force_replicate_table.t0 not exists for 2-th check, retry later table partition_table.t2 not exists for 4-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table ddl_sequence.finish_mark not exists for 13-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:54:06 GMT < Content-Length: 1792 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/147033ea-5040-4a9c-9c2e-cf88cf18795f [Mon May 6 19:54:07 CST 2024] <<<<<< START cdc server in event_filter case >>>>>> + [[ '' == \t\r\u\e ]] + set +e {"id":"147033ea-5040-4a9c-9c2e-cf88cf18795f","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996444} + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' /tidb/cdc/default/__cdc_meta__/capture/7a310c13-b8e5-4cb2-9d98-54c8055ead45 + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) {"id":"7a310c13-b8e5-4cb2-9d98-54c8055ead45","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996441} + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.event_filter.78897891.out server --log-file /tmp/tidb_cdc_test/event_filter/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/event_filter/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 /tidb/cdc/default/__cdc_meta__/capture/f72793a5-7503-4aca-8639-086db4a7bac6 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 {"id":"f72793a5-7503-4aca-8639-086db4a7bac6","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996439} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc08a6851 f72793a5-7503-4aca-8639-086db4a7bac6 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc08a687e 7a310c13-b8e5-4cb2-9d98-54c8055ead45 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc08a689f 147033ea-5040-4a9c-9c2e-cf88cf18795f /tidb/cdc/default/default/upstream/7365853528650801719 {"id":7365853528650801719,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/147033ea-5040-4a9c-9c2e-cf88cf18795f {"id":"147033ea-5040-4a9c-9c2e-cf88cf18795f","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996444} /tidb/cdc/default/__cdc_meta__/capture/7a310c13-b8e5-4cb2-9d98-54c8055ead45 {"id":"7a310c13-b8e5-4cb2-9d98-54c8055ead45","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996441} /tidb/cdc/default/__cdc_meta__/capture/f72793a5-7503-4aca-8639-086db4a7bac6 {"id":"f72793a5-7503-4aca-8639-086db4a7bac6","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996439} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc08a6851 f72793a5-7503-4aca-8639-086db4a7bac6 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc08a687e 7a310c13-b8e5-4cb2-9d98-54c8055ead45 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc08a689f 147033ea-5040-4a9c-9c2e-cf88cf18795f /tidb/cdc/default/default/upstream/7365853528650801719 {"id":7365853528650801719,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/147033ea-5040-4a9c-9c2e-cf88cf18795f {"id":"147033ea-5040-4a9c-9c2e-cf88cf18795f","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996444} /tidb/cdc/default/__cdc_meta__/capture/7a310c13-b8e5-4cb2-9d98-54c8055ead45 {"id":"7a310c13-b8e5-4cb2-9d98-54c8055ead45","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996441} /tidb/cdc/default/__cdc_meta__/capture/f72793a5-7503-4aca-8639-086db4a7bac6 {"id":"f72793a5-7503-4aca-8639-086db4a7bac6","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996439} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc08a6851 f72793a5-7503-4aca-8639-086db4a7bac6 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc08a687e 7a310c13-b8e5-4cb2-9d98-54c8055ead45 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc08a689f 147033ea-5040-4a9c-9c2e-cf88cf18795f /tidb/cdc/default/default/upstream/7365853528650801719 {"id":7365853528650801719,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_storage_partition_table.cli.10749.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/canal_json_storage_partition_table/storage_test?flush-interval=5s&enable-tidb-extension=true' --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_partition_table/conf/changefeed.toml Create changefeed successfully! ID: 280da8a8-ee96-4517-bb92-04ea2bdbbca9 Info: {"upstream_id":7365853528650801719,"namespace":"default","id":"280da8a8-ee96-4517-bb92-04ea2bdbbca9","sink_uri":"file:///tmp/tidb_cdc_test/canal_json_storage_partition_table/storage_test?flush-interval=5s\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:54:07.255392748+08:00","start_ts":449576028637495312,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576028637495312,"checkpoint_ts":449576028637495312,"checkpoint_time":"2024-05-06 19:54:07.134"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... table processor_delay.t8 not exists for 3-th check, retry later check diff failed 2-th time, retry later table force_replicate_table.t0 not exists for 3-th check, retry later + set +x table partition_table.t2 not exists for 5-th check, retry later table processor_delay.t8 not exists for 4-th check, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_sequence.finish_mark not exists for 14-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:54:10 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7c641c06-1f6b-4491-b0ba-ee782169670a {"id":"7c641c06-1f6b-4491-b0ba-ee782169670a","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996447} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc0c7f932 7c641c06-1f6b-4491-b0ba-ee782169670a /tidb/cdc/default/default/upstream/7365853595618831168 {"id":7365853595618831168,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7c641c06-1f6b-4491-b0ba-ee782169670a {"id":"7c641c06-1f6b-4491-b0ba-ee782169670a","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996447} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc0c7f932 7c641c06-1f6b-4491-b0ba-ee782169670a /tidb/cdc/default/default/upstream/7365853595618831168 {"id":7365853595618831168,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7c641c06-1f6b-4491-b0ba-ee782169670a {"id":"7c641c06-1f6b-4491-b0ba-ee782169670a","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996447} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc0c7f932 7c641c06-1f6b-4491-b0ba-ee782169670a /tidb/cdc/default/default/upstream/7365853595618831168 {"id":7365853595618831168,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.event_filter.cli.7943.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/event_filter/storage_test/ticdc-event-filter-24755?protocol=canal-json&enable-tidb-extension=true' --server=127.0.0.1:8300 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/event_filter/conf/cf.toml Create changefeed successfully! ID: a1830685-b5b6-4bff-9c06-3a314dd53416 Info: {"upstream_id":7365853595618831168,"namespace":"default","id":"a1830685-b5b6-4bff-9c06-3a314dd53416","sink_uri":"file:///tmp/tidb_cdc_test/event_filter/storage_test/ticdc-event-filter-24755?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:54:10.77539185+08:00","start_ts":449576029570990082,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["event_filter.*"],"event_filters":[{"matcher":["event_filter.t1"],"ignore_event":["drop table","delete"],"ignore_sql":null,"ignore_insert_value_expr":"id = 2 or city = 'tokyo'","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""},{"matcher":["event_filter.t_truncate"],"ignore_event":["truncate table"],"ignore_sql":null,"ignore_insert_value_expr":"","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""},{"matcher":["event_filter.t_alter"],"ignore_event":["alter table"],"ignore_sql":null,"ignore_insert_value_expr":"","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""}]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576029570990082,"checkpoint_ts":449576029570990082,"checkpoint_time":"2024-05-06 19:54:10.695"} PASS coverage: 2.8% of statements in github.com/pingcap/tiflow/... table partition_table.t2 not exists for 6-th check, retry later check diff failed 3-th time, retry later table force_replicate_table.t0 not exists for 4-th check, retry later table processor_delay.t8 not exists for 5-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_sequence.finish_mark not exists for 15-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/event_filter + sink_uri='file:///tmp/tidb_cdc_test/event_filter/storage_test/ticdc-event-filter-24755?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/event_filter ++ date + echo '[Mon May 6 19:54:12 CST 2024] <<<<<< START storage consumer in event_filter case >>>>>>' [Mon May 6 19:54:12 CST 2024] <<<<<< START storage consumer in event_filter case >>>>>> + cd /tmp/tidb_cdc_test/event_filter + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/event_filter + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/event_filter/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/event_filter/storage_test/ticdc-event-filter-24755?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' check diff failed 4-th time, retry later table force_replicate_table.t0 not exists for 5-th check, retry later table partition_table.t2 not exists for 7-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_sequence.finish_mark not exists for 16-th check, retry later table processor_delay.t8 exists table processor_delay.t9 exists table processor_delay.t10 not exists for 1-th check, retry later check diff failed 5-th time, retry later table force_replicate_table.t0 exists table force_replicate_table.t1 exists table force_replicate_table.t2 exists table force_replicate_table.t3 exists table force_replicate_table.t4 exists table force_replicate_table.t5 not exists for 1-th check, retry later table partition_table.t2 not exists for 8-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_sequence.finish_mark not exists for 17-th check, retry later table event_filter.t1 does not exists table event_filter.t1 not exists for 1-th check, retry later table processor_delay.t10 not exists for 2-th check, retry later table force_replicate_table.t5 not exists for 2-th check, retry later table partition_table.t2 not exists for 9-th check, retry later check diff failed 6-th time, retry later table ddl_sequence.finish_mark not exists for 18-th check, retry later table event_filter.t1 not exists for 2-th check, retry later table processor_delay.t10 not exists for 3-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370497f00006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-vs88z, pid:10644, start at 2024-05-06 19:54:14.913998246 +0800 CST m=+5.613630847 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:56:14.924 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:54:14.908 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:44:14.908 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370497f00006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-vs88z, pid:10644, start at 2024-05-06 19:54:14.913998246 +0800 CST m=+5.613630847 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:56:14.924 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:54:14.908 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:44:14.908 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370499700003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-vs88z, pid:10715, start at 2024-05-06 19:54:15.00710452 +0800 CST m=+5.655666174 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:56:15.018 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:54:15.004 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:44:15.004 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/csv_storage_partition_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/csv_storage_partition_table/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/csv_storage_partition_table/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/csv_storage_partition_table/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/csv_storage_partition_table/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table force_replicate_table.t5 not exists for 3-th check, retry later + workdir=/tmp/tidb_cdc_test/canal_json_storage_partition_table + sink_uri='file:///tmp/tidb_cdc_test/canal_json_storage_partition_table/storage_test?flush-interval=5s&enable-tidb-extension=true' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_partition_table/conf/changefeed.toml + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/canal_json_storage_partition_table ++ date + echo '[Mon May 6 19:54:18 CST 2024] <<<<<< START storage consumer in canal_json_storage_partition_table case >>>>>>' [Mon May 6 19:54:18 CST 2024] <<<<<< START storage consumer in canal_json_storage_partition_table case >>>>>> + cd /tmp/tidb_cdc_test/canal_json_storage_partition_table + '[' /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_partition_table/conf/changefeed.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/canal_json_storage_partition_table + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/canal_json_storage_partition_table/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/canal_json_storage_partition_table/storage_test?flush-interval=5s&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_partition_table/conf/changefeed.toml table partition_table.t2 not exists for 10-th check, retry later check diff failed 7-th time, retry later table processor_delay.t10 not exists for 4-th check, retry later [Mon May 6 19:54:19 CST 2024] <<<<<< START cdc server in csv_storage_partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_partition_table.1199011992.out server --log-file /tmp/tidb_cdc_test/csv_storage_partition_table/cdc0.log --log-level debug --data-dir /tmp/tidb_cdc_test/csv_storage_partition_table/cdc_data0 --cluster-id default --addr 127.0.0.1:8300 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_sequence.finish_mark not exists for 19-th check, retry later table event_filter.t1 not exists for 3-th check, retry later table partition_table.t2 not exists for 11-th check, retry later check diff failed 8-th time, retry later table force_replicate_table.t5 not exists for 4-th check, retry later table processor_delay.t10 not exists for 5-th check, retry later table ddl_sequence.finish_mark not exists for 20-th check, retry later TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 1-th time, retry later TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 2-th time, retry later TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 3-th time, retry later TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 4-th time, retry later table event_filter.t1 not exists for 4-th check, retry later table force_replicate_table.t5 not exists for 5-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:54:22 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e796e07b-28ce-4e76-ba93-000d5e556b5e {"id":"e796e07b-28ce-4e76-ba93-000d5e556b5e","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996460} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1001133 e796e07b-28ce-4e76-ba93-000d5e556b5e /tidb/cdc/default/default/upstream/7365853650940799577 {"id":7365853650940799577,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e796e07b-28ce-4e76-ba93-000d5e556b5e {"id":"e796e07b-28ce-4e76-ba93-000d5e556b5e","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996460} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1001133 e796e07b-28ce-4e76-ba93-000d5e556b5e /tidb/cdc/default/default/upstream/7365853650940799577 {"id":7365853650940799577,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e796e07b-28ce-4e76-ba93-000d5e556b5e {"id":"e796e07b-28ce-4e76-ba93-000d5e556b5e","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996460} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1001133 e796e07b-28ce-4e76-ba93-000d5e556b5e /tidb/cdc/default/default/upstream/7365853650940799577 {"id":7365853650940799577,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Mon May 6 19:54:22 CST 2024] <<<<<< START cdc server in csv_storage_partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_partition_table.1205112053.out server --log-file /tmp/tidb_cdc_test/csv_storage_partition_table/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/csv_storage_partition_table/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table partition_table.t2 not exists for 12-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 5-th time, retry later table event_filter.t1 exists table event_filter.t_normal exists table event_filter.t_truncate not exists for 1-th check, retry later table processor_delay.t10 exists table processor_delay.t11 exists table processor_delay.t12 not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... table ddl_sequence.finish_mark not exists for 21-th check, retry later wait process cdc.test exit for 3-th time... table force_replicate_table.t5 exists table force_replicate_table.t6 not exists for 1-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:54:24 CST 2024] <<<<<< run test case changefeed_pause_resume success! >>>>>> table partition_table.t2 not exists for 13-th check, retry later table event_filter.t_truncate not exists for 2-th check, retry later table processor_delay.t12 not exists for 2-th check, retry later table ddl_sequence.finish_mark not exists for 22-th check, retry later check data successfully wait process cdc.test exit for 1-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:54:25 GMT < Content-Length: 1299 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/53f4e0c4-c2b4-4225-b4d6-891fc50bc7a6 {"id":"53f4e0c4-c2b4-4225-b4d6-891fc50bc7a6","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996463} /tidb/cdc/default/__cdc_meta__/capture/e796e07b-28ce-4e76-ba93-000d5e556b5e {"id":"e796e07b-28ce-4e76-ba93-000d5e556b5e","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996460} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1001133 e796e07b-28ce-4e76-ba93-000d5e556b5e /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1001165 53f4e0c4-c2b4-4225-b4d6-891fc50bc7a6 /tidb/cdc/default/default/upstream/7365853650940799577 {"id":7365853650940799577,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/53f4e0c4-c2b4-4225-b4d6-891fc50bc7a6 {"id":"53f4e0c4-c2b4-4225-b4d6-891fc50bc7a6","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996463} /tidb/cdc/default/__cdc_meta__/capture/e796e07b-28ce-4e76-ba93-000d5e556b5e {"id":"e796e07b-28ce-4e76-ba93-000d5e556b5e","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996460} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1001133 e796e07b-28ce-4e76-ba93-000d5e556b5e /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1001165 53f4e0c4-c2b4-4225-b4d6-891fc50bc7a6 /tidb/cdc/default/default/upstream/7365853650940799577 {"id":7365853650940799577,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/53f4e0c4-c2b4-4225-b4d6-891fc50bc7a6 {"id":"53f4e0c4-c2b4-4225-b4d6-891fc50bc7a6","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996463} /tidb/cdc/default/__cdc_meta__/capture/e796e07b-28ce-4e76-ba93-000d5e556b5e {"id":"e796e07b-28ce-4e76-ba93-000d5e556b5e","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996460} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1001133 e796e07b-28ce-4e76-ba93-000d5e556b5e /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1001165 53f4e0c4-c2b4-4225-b4d6-891fc50bc7a6 /tidb/cdc/default/default/upstream/7365853650940799577 {"id":7365853650940799577,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Mon May 6 19:54:25 CST 2024] <<<<<< START cdc server in csv_storage_partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS= + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_partition_table.1211312115.out server --log-file /tmp/tidb_cdc_test/csv_storage_partition_table/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/csv_storage_partition_table/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 2-th time... table partition_table.t not exists for 1-th check, retry later table force_replicate_table.t6 not exists for 2-th check, retry later table partition_table.t2 not exists for 14-th check, retry later wait process cdc.test exit for 3-th time... table event_filter.t_truncate not exists for 3-th check, retry later table processor_delay.t12 not exists for 3-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:54:27 CST 2024] <<<<<< run test case ddl_puller_lag success! >>>>>> table ddl_sequence.finish_mark not exists for 23-th check, retry later table partition_table.t not exists for 2-th check, retry later table force_replicate_table.t6 not exists for 3-th check, retry later table partition_table.t2 not exists for 15-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:54:28 GMT < Content-Length: 1792 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/53f4e0c4-c2b4-4225-b4d6-891fc50bc7a6 {"id":"53f4e0c4-c2b4-4225-b4d6-891fc50bc7a6","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996463} /tidb/cdc/default/__cdc_meta__/capture/583cb59a-b2f1-4e9c-aceb-3514f7ab7dfd {"id":"583cb59a-b2f1-4e9c-aceb-3514f7ab7dfd","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996466} /tidb/cdc/default/__cdc_meta__/capture/e796e07b-28ce-4e76-ba93-000d5e556b5e {"id":"e796e07b-28ce-4e76-ba93-000d5e556b5e","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996460} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1001133 e796e07b-28ce-4e76-ba93-000d5e556b5e /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1001165 53f4e0c4-c2b4-4225-b4d6-891fc50bc7a6 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1001180 583cb59a-b2f1-4e9c-aceb-3514f7ab7dfd /tidb/cdc/default/default/upstream/7365853650940799577 {"id":7365853650940799577,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/53f4e0c4-c2b4-4225-b4d6-891fc50bc7a6 {"id":"53f4e0c4-c2b4-4225-b4d6-891fc50bc7a6","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996463} /tidb/cdc/default/__cdc_meta__/capture/583cb59a-b2f1-4e9c-aceb-3514f7ab7dfd {"id":"583cb59a-b2f1-4e9c-aceb-3514f7ab7dfd","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996466} /tidb/cdc/default/__cdc_meta__/capture/e796e07b-28ce-4e76-ba93-000d5e556b5e {"id":"e796e07b-28ce-4e76-ba93-000d5e556b5e","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996460} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1001133 e796e07b-28ce-4e76-ba93-000d5e556b5e /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1001165 53f4e0c4-c2b4-4225-b4d6-891fc50bc7a6 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1001180 583cb59a-b2f1-4e9c-aceb-3514f7ab7dfd /tidb/cdc/default/default/upstream/7365853650940799577 {"id":7365853650940799577,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/53f4e0c4-c2b4-4225-b4d6-891fc50bc7a6 {"id":"53f4e0c4-c2b4-4225-b4d6-891fc50bc7a6","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996463} /tidb/cdc/default/__cdc_meta__/capture/583cb59a-b2f1-4e9c-aceb-3514f7ab7dfd {"id":"583cb59a-b2f1-4e9c-aceb-3514f7ab7dfd","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996466} /tidb/cdc/default/__cdc_meta__/capture/e796e07b-28ce-4e76-ba93-000d5e556b5e {"id":"e796e07b-28ce-4e76-ba93-000d5e556b5e","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996460} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1001133 e796e07b-28ce-4e76-ba93-000d5e556b5e /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1001165 53f4e0c4-c2b4-4225-b4d6-891fc50bc7a6 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1001180 583cb59a-b2f1-4e9c-aceb-3514f7ab7dfd /tidb/cdc/default/default/upstream/7365853650940799577 {"id":7365853650940799577,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_partition_table.cli.12167.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/csv_storage_partition_table/storage_test?flush-interval=5s&enable-tidb-extension=true' --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_partition_table/conf/changefeed.toml Create changefeed successfully! ID: dacab7c4-959d-4cbe-badf-987ab544c513 Info: {"upstream_id":7365853650940799577,"namespace":"default","id":"dacab7c4-959d-4cbe-badf-987ab544c513","sink_uri":"file:///tmp/tidb_cdc_test/csv_storage_partition_table/storage_test?flush-interval=5s\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:54:29.320808328+08:00","start_ts":449576034424324102,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576034424324102,"checkpoint_ts":449576034424324102,"checkpoint_time":"2024-05-06 19:54:29.209"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... table processor_delay.t12 not exists for 4-th check, retry later table event_filter.t_truncate not exists for 4-th check, retry later table partition_table.t exists table ddl_sequence.finish_mark not exists for 24-th check, retry later table partition_table.t1 exists table partition_table.finish_mark not exists for 1-th check, retry later + set +x table partition_table.t2 exists table partition_table.finish_mark not exists for 1-th check, retry later table force_replicate_table.t6 not exists for 4-th check, retry later table event_filter.t_truncate not exists for 5-th check, retry later table processor_delay.t12 not exists for 5-th check, retry later table ddl_sequence.finish_mark not exists for 25-th check, retry later table partition_table.finish_mark not exists for 2-th check, retry later table force_replicate_table.t6 not exists for 5-th check, retry later table partition_table.finish_mark not exists for 2-th check, retry later table event_filter.t_truncate exists table event_filter.t_alter exists table processor_delay.t12 exists table processor_delay.t13 exists table processor_delay.t14 not exists for 1-th check, retry later table ddl_sequence.finish_mark not exists for 26-th check, retry later table force_replicate_table.t6 exists check_data_subset force_replicate_table.t0 127.0.0.1 4000 127.0.0.1 3306 table partition_table.finish_mark not exists for 3-th check, retry later table partition_table.finish_mark not exists for 3-th check, retry later run task successfully check_data_subset force_replicate_table.t1 127.0.0.1 4000 127.0.0.1 3306 table event_filter.finish_mark exists check diff failed 1-th time, retry later table processor_delay.t14 not exists for 2-th check, retry later run task successfully check_data_subset force_replicate_table.t2 127.0.0.1 4000 127.0.0.1 3306 table ddl_sequence.finish_mark not exists for 27-th check, retry later run task successfully check_data_subset force_replicate_table.t3 127.0.0.1 4000 127.0.0.1 3306 table partition_table.finish_mark not exists for 4-th check, retry later table partition_table.finish_mark not exists for 4-th check, retry later check diff failed 2-th time, retry later run task successfully check_data_subset force_replicate_table.t4 127.0.0.1 4000 127.0.0.1 3306 table processor_delay.t14 not exists for 3-th check, retry later table ddl_sequence.finish_mark exists run task successfully check_data_subset force_replicate_table.t5 127.0.0.1 4000 127.0.0.1 3306 check diff successfully wait process cdc.test exit for 1-th time... table partition_table.finish_mark not exists for 5-th check, retry later run task successfully check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 id=7,a=NULL doesn't exist in downstream table force_replicate_table.t6 run task failed 1-th time, retry later table partition_table.finish_mark not exists for 5-th check, retry later wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... check diff failed 3-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/cli_with_auth/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table processor_delay.t14 not exists for 4-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:54:40 CST 2024] <<<<<< run test case ddl_sequence success! >>>>>> table partition_table.finish_mark not exists for 6-th check, retry later check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 id=7,a=NULL doesn't exist in downstream table force_replicate_table.t6 run task failed 2-th time, retry later table partition_table.finish_mark not exists for 6-th check, retry later check diff failed 4-th time, retry later table processor_delay.t14 not exists for 5-th check, retry later table partition_table.finish_mark not exists for 7-th check, retry later table partition_table.finish_mark not exists for 7-th check, retry later check diff failed 5-th time, retry later table processor_delay.t14 exists table processor_delay.t15 exists table processor_delay.t16 not exists for 1-th check, retry later check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 id=7,a=NULL doesn't exist in downstream table force_replicate_table.t6 run task failed 3-th time, retry later table partition_table.finish_mark not exists for 8-th check, retry later table partition_table.finish_mark not exists for 8-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/ddl_only_block_related_table/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... check diff failed 6-th time, retry later table processor_delay.t16 not exists for 2-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/cli_with_auth Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... table partition_table.finish_mark not exists for 9-th check, retry later table partition_table.finish_mark not exists for 9-th check, retry later check diff failed 7-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/ddl_only_block_related_table Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... table processor_delay.t16 not exists for 3-th check, retry later table partition_table.finish_mark not exists for 10-th check, retry later table partition_table.finish_mark not exists for 10-th check, retry later table processor_delay.t16 not exists for 4-th check, retry later check diff failed 8-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table partition_table.finish_mark not exists for 11-th check, retry later table partition_table.finish_mark not exists for 11-th check, retry later check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 id=7,a=NULL doesn't exist in downstream table force_replicate_table.t6 run task failed 4-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/resourcecontrol/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... check diff failed 9-th time, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.finish_mark not exists for 12-th check, retry later table processor_delay.t16 not exists for 5-th check, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.finish_mark not exists for 12-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t16 exists table processor_delay.t17 exists table processor_delay.t18 not exists for 1-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... table partition_table.finish_mark not exists for 13-th check, retry later table partition_table.finish_mark not exists for 13-th check, retry later wait process cdc.test exit for 2-th time... start tidb cluster in /tmp/tidb_cdc_test/resourcecontrol Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 3-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t18 not exists for 2-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:54:56 CST 2024] <<<<<< run test case event_filter success! >>>>>> Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table partition_table.finish_mark not exists for 14-th check, retry later table partition_table.finish_mark not exists for 14-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t18 not exists for 3-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.finish_mark not exists for 15-th check, retry later table partition_table.finish_mark exists check diff successfully check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 run task successfully VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370742680016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-nrrkt, pid:6721, start at 2024-05-06 19:54:58.635436023 +0800 CST m=+5.301770881 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:56:58.644 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:54:58.636 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:44:58.636 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... wait process cdc.test exit for 1-th time... table processor_delay.t18 not exists for 4-th check, retry later wait process cdc.test exit for 2-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370731900013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz, pid:12036, start at 2024-05-06 19:54:57.555208703 +0800 CST m=+5.330009862 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:56:57.562 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:54:57.558 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:44:57.558 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370731900013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz, pid:12036, start at 2024-05-06 19:54:57.555208703 +0800 CST m=+5.330009862 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:56:57.562 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:54:57.558 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:44:57.558 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3707361c0015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz, pid:12119, start at 2024-05-06 19:54:57.839421745 +0800 CST m=+5.543141804 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:56:57.847 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:54:57.849 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:44:57.849 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cli_with_auth/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cli_with_auth/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/cli_with_auth/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/cli_with_auth/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cli_with_auth/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 2-th time... table partition_table.finish_mark not exists for 16-th check, retry later wait process cdc.test exit for 3-th time... wait process cdc.test exit for 3-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:55:01 CST 2024] <<<<<< run test case partition_table success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370742680016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-nrrkt, pid:6721, start at 2024-05-06 19:54:58.635436023 +0800 CST m=+5.301770881 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:56:58.644 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:54:58.636 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:44:58.636 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370744a80007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-nrrkt, pid:6805, start at 2024-05-06 19:54:58.736714985 +0800 CST m=+5.348204225 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:56:58.745 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:54:58.730 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:44:58.730 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:55:01 CST 2024] <<<<<< run test case force_replicate_table success! >>>>>> + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.13392.out cli tso query --pd=http://127.0.0.1:2379 table partition_table.finish_mark exists table processor_delay.t18 not exists for 5-th check, retry later + workdir=/tmp/tidb_cdc_test/csv_storage_partition_table + sink_uri='file:///tmp/tidb_cdc_test/csv_storage_partition_table/storage_test?flush-interval=5s&enable-tidb-extension=true' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_partition_table/conf/changefeed.toml + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/csv_storage_partition_table ++ date + echo '[Mon May 6 19:55:01 CST 2024] <<<<<< START storage consumer in csv_storage_partition_table case >>>>>>' [Mon May 6 19:55:01 CST 2024] <<<<<< START storage consumer in csv_storage_partition_table case >>>>>> + cd /tmp/tidb_cdc_test/csv_storage_partition_table + '[' /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_partition_table/conf/changefeed.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/csv_storage_partition_table + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/csv_storage_partition_table/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/csv_storage_partition_table/storage_test?flush-interval=5s&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_partition_table/conf/changefeed.toml check diff successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... [Mon May 6 19:55:03 CST 2024] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.80558057.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 3-th time... + set +x + tso='449576043271421953 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576043271421953 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x table processor_delay.t18 exists table processor_delay.t19 exists table processor_delay.t20 not exists for 1-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:55:04 CST 2024] <<<<<< run test case canal_json_storage_partition_table success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370798e40004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-drnnn, pid:10480, start at 2024-05-06 19:55:04.122284326 +0800 CST m=+5.313292216 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:57:04.130 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:55:04.121 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:45:04.121 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Mon May 6 19:55:05 CST 2024] <<<<<< START cdc server in cli_with_auth case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.1344113443.out server --log-file /tmp/tidb_cdc_test/cli_with_auth/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cli_with_auth/cdc_data --cluster-id default --config /tmp/tidb_cdc_test/cli_with_auth/server.toml + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t20 not exists for 2-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370798e40004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-drnnn, pid:10480, start at 2024-05-06 19:55:04.122284326 +0800 CST m=+5.313292216 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:57:04.130 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:55:04.121 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:45:04.121 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d37079998000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-drnnn, pid:10554, start at 2024-05-06 19:55:04.179685849 +0800 CST m=+5.319163626 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:57:04.188 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:55:04.166 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:45:04.166 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/resourcecontrol/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/resourcecontrol/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:55:06 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/15fbe829-2d83-4c4e-97d4-7f23713d6f9a {"id":"15fbe829-2d83-4c4e-97d4-7f23713d6f9a","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996504} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1a82834 15fbe829-2d83-4c4e-97d4-7f23713d6f9a /tidb/cdc/default/default/upstream/7365853838014817680 {"id":7365853838014817680,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/15fbe829-2d83-4c4e-97d4-7f23713d6f9a {"id":"15fbe829-2d83-4c4e-97d4-7f23713d6f9a","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996504} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1a82834 15fbe829-2d83-4c4e-97d4-7f23713d6f9a /tidb/cdc/default/default/upstream/7365853838014817680 {"id":7365853838014817680,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/15fbe829-2d83-4c4e-97d4-7f23713d6f9a {"id":"15fbe829-2d83-4c4e-97d4-7f23713d6f9a","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996504} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1a82834 15fbe829-2d83-4c4e-97d4-7f23713d6f9a /tidb/cdc/default/default/upstream/7365853838014817680 {"id":7365853838014817680,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.cli.8114.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-10137?protocol=canal-json&enable-tidb-extension=true' -c=ddl-only-block-related-table Create changefeed successfully! ID: ddl-only-block-related-table Info: {"upstream_id":7365853838014817680,"namespace":"default","id":"ddl-only-block-related-table","sink_uri":"file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-10137?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:55:07.212378848+08:00","start_ts":449576044353552388,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576044353552388,"checkpoint_ts":449576044353552388,"checkpoint_time":"2024-05-06 19:55:07.086"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... table processor_delay.t20 not exists for 3-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:55:08 GMT < Content-Length: 873 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a57a6cd9-a406-4b14-8e72-2f2305cb36af {"id":"a57a6cd9-a406-4b14-8e72-2f2305cb36af","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996505} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1a3ba89 a57a6cd9-a406-4b14-8e72-2f2305cb36af /tidb/cdc/default/default/upstream/7365853834131847599 {"id":7365853834131847599,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a57a6cd9-a406-4b14-8e72-2f2305cb36af {"id":"a57a6cd9-a406-4b14-8e72-2f2305cb36af","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996505} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1a3ba89 a57a6cd9-a406-4b14-8e72-2f2305cb36af /tidb/cdc/default/default/upstream/7365853834131847599 {"id":7365853834131847599,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a57a6cd9-a406-4b14-8e72-2f2305cb36af {"id":"a57a6cd9-a406-4b14-8e72-2f2305cb36af","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996505} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1a3ba89 a57a6cd9-a406-4b14-8e72-2f2305cb36af /tidb/cdc/default/default/upstream/7365853834131847599 {"id":7365853834131847599,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.13498.out cli changefeed create --start-ts=449576043271421953 '--sink-uri=file:///tmp/tidb_cdc_test/cli_with_auth/storage_test/ticdc-cli-test-6289?protocol=canal-json&enable-tidb-extension=true' --tz=Asia/Shanghai -c=custom-changefeed-name + set +x + workdir=/tmp/tidb_cdc_test/ddl_only_block_related_table + sink_uri='file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-10137?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/ddl_only_block_related_table ++ date + echo '[Mon May 6 19:55:08 CST 2024] <<<<<< START storage consumer in ddl_only_block_related_table case >>>>>>' [Mon May 6 19:55:08 CST 2024] <<<<<< START storage consumer in ddl_only_block_related_table case >>>>>> + cd /tmp/tidb_cdc_test/ddl_only_block_related_table + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/ddl_only_block_related_table + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-10137?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' [WARN] --tz is deprecated in changefeed settings. + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.cli.11882.out cli tso query --pd=http://127.0.0.1:2379 Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7365853834131847599,"namespace":"default","id":"custom-changefeed-name","sink_uri":"file:///tmp/tidb_cdc_test/cli_with_auth/storage_test/ticdc-cli-test-6289?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:55:09.011255715+08:00","start_ts":449576043271421953,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576043271421953,"checkpoint_ts":449576043271421953,"checkpoint_time":"2024-05-06 19:55:02.958"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... table ddl_only_block_related_table.finish_mark not exists for 1-th check, retry later table processor_delay.t20 not exists for 4-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/cli_with_auth + sink_uri='file:///tmp/tidb_cdc_test/cli_with_auth/storage_test/ticdc-cli-test-6289?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/cli_with_auth ++ date + echo '[Mon May 6 19:55:10 CST 2024] <<<<<< START storage consumer in cli_with_auth case >>>>>>' [Mon May 6 19:55:10 CST 2024] <<<<<< START storage consumer in cli_with_auth case >>>>>> + cd /tmp/tidb_cdc_test/cli_with_auth + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/cli_with_auth + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/cli_with_auth/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/cli_with_auth/storage_test/ticdc-cli-test-6289?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table test.simple not exists for 1-th check, retry later table partition_table.t not exists for 1-th check, retry later + set +x + tso='449576044965920769 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576044965920769 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:55:10 CST 2024] <<<<<< START cdc server in resourcecontrol case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.1191811920.out server --log-file /tmp/tidb_cdc_test/resourcecontrol/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/resourcecontrol/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/sql_mode/run.sh using Sink-Type: storage... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/sql_mode/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/sql_mode ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/sql_mode + source /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/sql_mode/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/sql_mode + CDC_BINARY=cdc.test + SINK_TYPE=storage + CDC_COUNT=3 + DB_COUNT=4 + rm -rf /tmp/tidb_cdc_test/sql_mode + mkdir -p /tmp/tidb_cdc_test/sql_mode + start_tidb_cluster --workdir /tmp/tidb_cdc_test/sql_mode The 1 times to try to start tidb cluster... table ddl_only_block_related_table.finish_mark not exists for 2-th check, retry later table partition_table.t not exists for 2-th check, retry later table processor_delay.t20 not exists for 5-th check, retry later table test.simple not exists for 2-th check, retry later table ddl_only_block_related_table.finish_mark not exists for 3-th check, retry later table partition_table.t exists table partition_table.t1 exists table partition_table.finish_mark not exists for 1-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container table processor_delay.t20 exists table processor_delay.t21 exists table processor_delay.t22 not exists for 1-th check, retry later [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node table test.simple not exists for 3-th check, retry later [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:55:14 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0de63f10-1eee-49ba-b3bf-789d6bf78a41 {"id":"0de63f10-1eee-49ba-b3bf-789d6bf78a41","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996511} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1c36f3a 0de63f10-1eee-49ba-b3bf-789d6bf78a41 /tidb/cdc/default/default/upstream/7365853861389001317 {"id":7365853861389001317,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0de63f10-1eee-49ba-b3bf-789d6bf78a41 {"id":"0de63f10-1eee-49ba-b3bf-789d6bf78a41","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996511} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1c36f3a 0de63f10-1eee-49ba-b3bf-789d6bf78a41 /tidb/cdc/default/default/upstream/7365853861389001317 {"id":7365853861389001317,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0de63f10-1eee-49ba-b3bf-789d6bf78a41 {"id":"0de63f10-1eee-49ba-b3bf-789d6bf78a41","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996511} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1c36f3a 0de63f10-1eee-49ba-b3bf-789d6bf78a41 /tidb/cdc/default/default/upstream/7365853861389001317 {"id":7365853861389001317,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.cli.11977.out cli changefeed create --start-ts=449576044965920769 '--sink-uri=file:///tmp/tidb_cdc_test/resourcecontrol/storage_test/ticdc-resourcecontrol-test-25853?protocol=canal-json&enable-tidb-extension=true' [Pipeline] } [Pipeline] // stage [Pipeline] } Create changefeed successfully! ID: 5a315bcc-ceaf-470b-9d18-286f11c3042c Info: {"upstream_id":7365853861389001317,"namespace":"default","id":"5a315bcc-ceaf-470b-9d18-286f11c3042c","sink_uri":"file:///tmp/tidb_cdc_test/resourcecontrol/storage_test/ticdc-resourcecontrol-test-25853?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:55:15.316522468+08:00","start_ts":449576044965920769,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576044965920769,"checkpoint_ts":449576044965920769,"checkpoint_time":"2024-05-06 19:55:09.422"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... table ddl_only_block_related_table.finish_mark not exists for 4-th check, retry later table partition_table.finish_mark not exists for 2-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } table processor_delay.t22 not exists for 2-th check, retry later [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } + set +x + workdir=/tmp/tidb_cdc_test/resourcecontrol + sink_uri='file:///tmp/tidb_cdc_test/resourcecontrol/storage_test/ticdc-resourcecontrol-test-25853?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/resourcecontrol ++ date + echo '[Mon May 6 19:55:16 CST 2024] <<<<<< START storage consumer in resourcecontrol case >>>>>>' [Mon May 6 19:55:16 CST 2024] <<<<<< START storage consumer in resourcecontrol case >>>>>> + cd /tmp/tidb_cdc_test/resourcecontrol + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/resourcecontrol + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/resourcecontrol/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/resourcecontrol/storage_test/ticdc-resourcecontrol-test-25853?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table test.simple not exists for 4-th check, retry later [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table resourcecontrol.finish_mark not exists for 1-th check, retry later table ddl_only_block_related_table.finish_mark not exists for 5-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/sql_mode Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... table partition_table.finish_mark not exists for 3-th check, retry later table processor_delay.t22 not exists for 3-th check, retry later table test.simple not exists for 5-th check, retry later table ddl_only_block_related_table.finish_mark not exists for 6-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/multi_tables_ddl/run.sh using Sink-Type: storage... <<================= * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:24927; Connection refused * Closing connection 0 You are running an older version of MinIO released 3 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide table resourcecontrol.finish_mark not exists for 2-th check, retry later table partition_table.finish_mark not exists for 4-th check, retry later * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:24927 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 17CCE43FAC32C663 < X-Xss-Protection: 1; mode=block < Date: Mon, 06 May 2024 11:55:20 GMT < { [data not shown] * Connection #0 to host 127.0.0.1 left intact table processor_delay.t22 not exists for 4-th check, retry later Bucket 's3://logbucket/' created [Mon May 6 19:55:20 CST 2024] <<<<<< run test case multi_tables_ddl success! >>>>>> Exiting on signal: INTERRUPT table test.simple not exists for 6-th check, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table resourcecontrol.finish_mark not exists for 3-th check, retry later table ddl_only_block_related_table.finish_mark exists wait process 8060 exit for 1-th time... table partition_table.finish_mark not exists for 5-th check, retry later wait process 8060 exit for 2-th time... table processor_delay.t22 not exists for 5-th check, retry later wait process 8060 exit for 3-th time... table test.simple exists table test.`simple-dash` exists + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7365853834131847599, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449576048082026501, "checkpoint_time": "2024-05-06 19:55:21.309", "error": null }' + echo '{ "upstream_id": 7365853834131847599, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449576048082026501, "checkpoint_time": "2024-05-06 19:55:21.309", "error": null }' { "upstream_id": 7365853834131847599, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449576048082026501, "checkpoint_time": "2024-05-06 19:55:21.309", "error": null } ++ echo '{' '"upstream_id":' 7365853834131847599, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449576048082026501, '"checkpoint_time":' '"2024-05-06' '19:55:21.309",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7365853834131847599, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449576048082026501, '"checkpoint_time":' '"2024-05-06' '19:55:21.309",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] changefeed count 1 check pass, pd_addr: http://127.0.0.1:2379 changefeed count 1 check pass, pd_addr: http://127.0.0.1:2679 changefeed count 1 check pass, pd_addr: http://127.0.0.1:2779 /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (8060) - No such process wait process 8060 exit for 4-th time... process 8060 already exit [Mon May 6 19:55:23 CST 2024] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteNotDone=return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.82498251.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 changefeed count 1 check pass, pd_addr: http://127.0.0.1:2379,http://127.0.0.1:2679,http://127.0.0.1:2779 table resourcecontrol.finish_mark not exists for 4-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.finish_mark not exists for 6-th check, retry later Error: [CDC:ErrChangefeedUpdateRefused]changefeed update error: can only update changefeed config when it is stopped or failed update changefeed config should fail when changefeed is running, got Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config SyncPointInterval] From: To:0xc003f03010} {Type:update Path:[Config SyncPointRetention] From: To:0xc003f03018} {Type:update Path:[Config Consistent] From: To:0xc001306cb0} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.13926.out cli changefeed --changefeed-id custom-changefeed-name pause \033[0;36m<<< Run all test success >>>\033[0m table processor_delay.t22 exists table processor_delay.t23 exists table processor_delay.t24 not exists for 1-th check, retry later [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } table resourcecontrol.finish_mark not exists for 5-th check, retry later [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.finish_mark not exists for 7-th check, retry later [Pipeline] // stage [Pipeline] } + set +x table processor_delay.t24 not exists for 2-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:55:26 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7365853838014817680 Namespace:default ID:ddl-only-block-related-table SinkURI:file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-10137?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:55:07.212378848 +0800 CST StartTs:449576044353552388 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003b7efc0 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449576044379766790} {CheckpointTs:449576047840067588 MinTableBarrierTs:449576049150787588 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449576047840067588, checkpointTs: 449576047840067588, state: Preparing span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449576047840067588, checkpointTs: 449576047840067588, state: Preparing span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449576047840067588, checkpointTs: 449576047840067588, state: Preparing span: {table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}, resolvedTs: 449576047840067588, checkpointTs: 449576047840067588, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/27a4049f-1c11-496b-a8ed-bd0fa6fb0b4e {"id":"27a4049f-1c11-496b-a8ed-bd0fa6fb0b4e","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996523} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1a8291d 27a4049f-1c11-496b-a8ed-bd0fa6fb0b4e /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7365853838014817680,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-10137?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:55:07.212378848+08:00","start-ts":449576044353552388,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449576044379766790} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":449576047840067588,"min-table-barrier-ts":449576049150787588,"admin-job-type":0} /tidb/cdc/default/default/task/position/27a4049f-1c11-496b-a8ed-bd0fa6fb0b4e/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365853838014817680 {"id":7365853838014817680,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7365853838014817680 Namespace:default ID:ddl-only-block-related-table SinkURI:file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-10137?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:55:07.212378848 +0800 CST StartTs:449576044353552388 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003b7efc0 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449576044379766790} {CheckpointTs:449576047840067588 MinTableBarrierTs:449576049150787588 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449576047840067588, checkpointTs: 449576047840067588, state: Preparing span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449576047840067588, checkpointTs: 449576047840067588, state: Preparing span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449576047840067588, checkpointTs: 449576047840067588, state: Preparing span: {table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}, resolvedTs: 449576047840067588, checkpointTs: 449576047840067588, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/27a4049f-1c11-496b-a8ed-bd0fa6fb0b4e {"id":"27a4049f-1c11-496b-a8ed-bd0fa6fb0b4e","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996523} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1a8291d 27a4049f-1c11-496b-a8ed-bd0fa6fb0b4e /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7365853838014817680,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-10137?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:55:07.212378848+08:00","start-ts":449576044353552388,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449576044379766790} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":449576047840067588,"min-table-barrier-ts":449576049150787588,"admin-job-type":0} /tidb/cdc/default/default/task/position/27a4049f-1c11-496b-a8ed-bd0fa6fb0b4e/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365853838014817680 {"id":7365853838014817680,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7365853838014817680 Namespace:default ID:ddl-only-block-related-table SinkURI:file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-10137?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:55:07.212378848 +0800 CST StartTs:449576044353552388 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003b7efc0 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449576044379766790} {CheckpointTs:449576047840067588 MinTableBarrierTs:449576049150787588 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449576047840067588, checkpointTs: 449576047840067588, state: Preparing span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449576047840067588, checkpointTs: 449576047840067588, state: Preparing span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449576047840067588, checkpointTs: 449576047840067588, state: Preparing span: {table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}, resolvedTs: 449576047840067588, checkpointTs: 449576047840067588, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/27a4049f-1c11-496b-a8ed-bd0fa6fb0b4e {"id":"27a4049f-1c11-496b-a8ed-bd0fa6fb0b4e","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996523} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1a8291d 27a4049f-1c11-496b-a8ed-bd0fa6fb0b4e /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7365853838014817680,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-10137?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:55:07.212378848+08:00","start-ts":449576044353552388,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449576044379766790} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":449576047840067588,"min-table-barrier-ts":449576049150787588,"admin-job-type":0} /tidb/cdc/default/default/task/position/27a4049f-1c11-496b-a8ed-bd0fa6fb0b4e/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365853838014817680 {"id":7365853838014817680,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x check_ts_not_forward ddl-only-block-related-table table resourcecontrol.finish_mark exists check diff successfully VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3708f4f40014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3, pid:9412, start at 2024-05-06 19:55:26.445668808 +0800 CST m=+5.258375381 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:57:26.452 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:55:26.446 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:45:26.446 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... table partition_table.finish_mark not exists for 8-th check, retry later wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... process cdc.test already exit [Mon May 6 19:55:28 CST 2024] <<<<<< run test case resourcecontrol success! >>>>>> table processor_delay.t24 not exists for 3-th check, retry later run task failed 1-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3708f4f40014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3, pid:9412, start at 2024-05-06 19:55:26.445668808 +0800 CST m=+5.258375381 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:57:26.452 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:55:26.446 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:45:26.446 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3708f6640017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3, pid:9485, start at 2024-05-06 19:55:26.536664071 +0800 CST m=+5.292337636 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:57:26.543 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:55:26.538 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:45:26.538 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sql_mode/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sql_mode/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/sql_mode/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sql_mode/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/sql_mode/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table partition_table.finish_mark not exists for 9-th check, retry later + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=stopped + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7365853834131847599, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 449576048868196357, "checkpoint_time": "2024-05-06 19:55:24.308", "error": null }' + echo '{ "upstream_id": 7365853834131847599, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 449576048868196357, "checkpoint_time": "2024-05-06 19:55:24.308", "error": null }' { "upstream_id": 7365853834131847599, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 449576048868196357, "checkpoint_time": "2024-05-06 19:55:24.308", "error": null } ++ echo '{' '"upstream_id":' 7365853834131847599, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 449576048868196357, '"checkpoint_time":' '"2024-05-06' '19:55:24.308",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7365853834131847599, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 449576048868196357, '"checkpoint_time":' '"2024-05-06' '19:55:24.308",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.14017.out cli changefeed update --pd=http://127.0.0.1:2379,http://127.0.0.1:2679,http://127.0.0.1:2779 --config=/tmp/tidb_cdc_test/cli_with_auth/changefeed.toml --no-confirm --changefeed-id custom-changefeed-name Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config SyncPointInterval] From: To:0xc0038c8278} {Type:update Path:[Config SyncPointRetention] From: To:0xc0038c8288} {Type:update Path:[Config Consistent] From: To:0xc000a2a4d0} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} Update changefeed config successfully! ID: custom-changefeed-name Info: {"upstream_id":7365853834131847599,"namespace":"default","id":"custom-changefeed-name","sink_uri":"file:///tmp/tidb_cdc_test/cli_with_auth/storage_test/ticdc-cli-test-6289?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:55:09.011255715+08:00","start_ts":449576043271421953,"admin_job_type":1,"config":{"memory_quota":1073741824,"case_sensitive":true,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_table_monitor":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"delete_only_output_handle_key_columns":null,"content_compatible":null,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"stopped","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":0,"checkpoint_ts":449576048868196357,"checkpoint_time":"2024-05-06 19:55:24.308"} PASS coverage: 3.0% of statements in github.com/pingcap/tiflow/... table processor_delay.t24 not exists for 4-th check, retry later check_ts_not_forward ddl-only-block-related-table + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.14058.out cli changefeed --changefeed-id custom-changefeed-name resume + trap stop_tidb_cluster EXIT + run_sql 'set global sql_mode='\''NO_BACKSLASH_ESCAPES'\'';' 127.0.0.1 4000 + run_sql 'set global sql_mode='\''NO_BACKSLASH_ESCAPES'\'';' 127.0.0.1 3306 + cd /tmp/tidb_cdc_test/sql_mode ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.cli.10875.out cli tso query --pd=http://127.0.0.1:2379 table partition_table.finish_mark not exists for 10-th check, retry later PASS coverage: 2.3% of statements in github.com/pingcap/tiflow/... table processor_delay.t24 not exists for 5-th check, retry later + set +x + tso='449576050844499971 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576050844499971 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449576050844499971 + run_cdc_server --workdir /tmp/tidb_cdc_test/sql_mode --binary cdc.test [Mon May 6 19:55:33 CST 2024] <<<<<< START cdc server in sql_mode case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.1091510917.out server --log-file /tmp/tidb_cdc_test/sql_mode/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sql_mode/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table partition_table.finish_mark not exists for 11-th check, retry later + set +x table processor_delay.t24 exists table processor_delay.t25 exists table processor_delay.t26 not exists for 1-th check, retry later table partition_table.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:55:36 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2c730f9d-d7fb-4209-9d0d-b99184cd4c96 {"id":"2c730f9d-d7fb-4209-9d0d-b99184cd4c96","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996533} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc21ad73f 2c730f9d-d7fb-4209-9d0d-b99184cd4c96 /tidb/cdc/default/default/upstream/7365853961037049289 {"id":7365853961037049289,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2c730f9d-d7fb-4209-9d0d-b99184cd4c96 {"id":"2c730f9d-d7fb-4209-9d0d-b99184cd4c96","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996533} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc21ad73f 2c730f9d-d7fb-4209-9d0d-b99184cd4c96 /tidb/cdc/default/default/upstream/7365853961037049289 {"id":7365853961037049289,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2c730f9d-d7fb-4209-9d0d-b99184cd4c96 {"id":"2c730f9d-d7fb-4209-9d0d-b99184cd4c96","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996533} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc21ad73f 2c730f9d-d7fb-4209-9d0d-b99184cd4c96 /tidb/cdc/default/default/upstream/7365853961037049289 {"id":7365853961037049289,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449576050844499971 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.cli.10970.out cli changefeed create --start-ts=449576050844499971 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 Create changefeed successfully! ID: test-1 Info: {"upstream_id":7365853961037049289,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-06T19:55:36.88993632+08:00","start_ts":449576050844499971,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576050844499971,"checkpoint_ts":449576050844499971,"checkpoint_time":"2024-05-06 19:55:31.847"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... table processor_delay.t26 not exists for 2-th check, retry later + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7365853834131847599, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449576052014186503, "checkpoint_time": "2024-05-06 19:55:36.309", "error": null }' + echo '{ "upstream_id": 7365853834131847599, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449576052014186503, "checkpoint_time": "2024-05-06 19:55:36.309", "error": null }' { "upstream_id": 7365853834131847599, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449576052014186503, "checkpoint_time": "2024-05-06 19:55:36.309", "error": null } ++ echo '{' '"upstream_id":' 7365853834131847599, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449576052014186503, '"checkpoint_time":' '"2024-05-06' '19:55:36.309",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7365853834131847599, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449576052014186503, '"checkpoint_time":' '"2024-05-06' '19:55:36.309",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.14147.out cli changefeed --changefeed-id custom-changefeed-name remove wait process cdc.test exit for 3-th time... Changefeed remove successfully. ID: custom-changefeed-name CheckpointTs: 449576052014186503 SinkURI: file:///tmp/tidb_cdc_test/cli_with_auth/storage_test/ticdc-cli-test-6289?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:55:38 CST 2024] <<<<<< run test case csv_storage_partition_table success! >>>>>> + set +x + run_sql 'use test; create table t1(id bigint primary key, a text, b text as ((regexp_replace(a, '\''^[1-9]\d{9,29}$'\'', '\''aaaaa'\''))), c text); insert into t1 (id, a, c) values(1,123456, '\''ab\\\\c'\''); insert into t1 (id, a, c) values(2,1234567890123, '\''ab\\c'\'');' 127.0.0.1 4000 + '[' storage == mysql ']' + stop_tidb_cluster table processor_delay.t26 not exists for 3-th check, retry later + set +x [2024/05/06 19:55:33.304 +08:00] [INFO] [pd_service_discovery.go:1016] ["[pd] switch leader"] [new-leader=http://127.0.0.1:2379] [old-leader=] [2024/05/06 19:55:33.304 +08:00] [INFO] [pd_service_discovery.go:498] ["[pd] init cluster id"] [cluster-id=7365852987147108412] [2024/05/06 19:55:33.305 +08:00] [INFO] [client.go:606] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2024/05/06 19:55:33.305 +08:00] [INFO] [tso_client.go:236] ["[tso] switch dc tso global allocator serving url"] [dc-location=global] [new-url=http://127.0.0.1:2379] [2024/05/06 19:55:33.306 +08:00] [INFO] [tso_dispatcher.go:359] ["[tso] tso dispatcher created"] [dc-location=global] [2024/05/06 19:55:33.306 +08:00] [INFO] [client.go:612] ["[pd] service mode changed"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2024/05/06 19:55:33.308 +08:00] [INFO] [pd_service_discovery.go:1016] ["[pd] switch leader"] [new-leader=http://127.0.0.1:2379] [old-leader=] [2024/05/06 19:55:33.308 +08:00] [INFO] [pd_service_discovery.go:498] ["[pd] init cluster id"] [cluster-id=7365852987147108412] [2024/05/06 19:55:33.308 +08:00] [INFO] [client.go:606] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2024/05/06 19:55:33.308 +08:00] [INFO] [tso_client.go:236] ["[tso] switch dc tso global allocator serving url"] [dc-location=global] [new-url=http://127.0.0.1:2379] [2024/05/06 19:55:33.309 +08:00] [INFO] [tso_dispatcher.go:359] ["[tso] tso dispatcher created"] [dc-location=global] [2024/05/06 19:55:33.309 +08:00] [INFO] [client.go:612] ["[pd] service mode changed"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2024/05/06 19:55:33.317 +08:00] [INFO] [tikv_driver.go:197] ["using API V1."] [2024/05/06 19:55:33.319 +08:00] [INFO] [main.go:180] ["genLock started"] [2024/05/06 19:55:33.324 +08:00] [INFO] [store_cache.go:477] ["change store resolve state"] [store=1] [addr=127.0.0.1:20161] [from=unresolved] [to=resolved] [liveness-state=reachable] table processor_delay.t26 not exists for 4-th check, retry later changefeed count 0 check pass, pd_addr: http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.14228.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/cli_with_auth/storage_test/ticdc-cli-test-6289?protocol=canal-json&enable-tidb-extension=true' --tz=Asia/Shanghai -c=custom-changefeed-name table processor_delay.t26 not exists for 5-th check, retry later [WARN] --tz is deprecated in changefeed settings. Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7365853834131847599,"namespace":"default","id":"custom-changefeed-name","sink_uri":"file:///tmp/tidb_cdc_test/cli_with_auth/storage_test/ticdc-cli-test-6289?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:55:42.958754112+08:00","start_ts":449576053717860358,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576053717860358,"checkpoint_ts":449576053717860358,"checkpoint_time":"2024-05-06 19:55:42.808"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... [2024/05/06 19:55:43.323 +08:00] [INFO] [main.go:196] ["genLock done"] [2024/05/06 19:55:43.323 +08:00] [INFO] [pd_service_discovery.go:550] ["[pd] exit member loop due to context canceled"] [2024/05/06 19:55:43.323 +08:00] [INFO] [tso_dispatcher.go:455] ["[tso] stop fetching the pending tso requests due to context canceled"] [dc-location=global] [2024/05/06 19:55:43.323 +08:00] [INFO] [tso_dispatcher.go:380] ["[tso] exit tso dispatcher"] [dc-location=global] [2024/05/06 19:55:43.323 +08:00] [INFO] [tso_batch_controller.go:158] ["[pd] clear the tso batch controller"] [max-batch-size=10000] [best-batch-size=1] [collected-request-count=0] [pending-request-count=0] [2024/05/06 19:55:43.323 +08:00] [INFO] [tso_dispatcher.go:214] ["exit tso requests cancel loop"] [2024/05/06 19:55:43.323 +08:00] [INFO] [tso_dispatcher.go:268] ["exit tso dispatcher loop"] [2024/05/06 19:55:43.323 +08:00] [INFO] [resource_manager_client.go:295] ["[resource manager] exit resource token dispatcher"] [2024/05/06 19:55:43.323 +08:00] [INFO] [tso_client.go:140] ["closing tso client"] [2024/05/06 19:55:43.323 +08:00] [INFO] [tso_client.go:145] ["close tso client"] [2024/05/06 19:55:43.323 +08:00] [INFO] [tso_batch_controller.go:158] ["[pd] clear the tso batch controller"] [max-batch-size=10000] [best-batch-size=1] [collected-request-count=0] [pending-request-count=0] [2024/05/06 19:55:43.323 +08:00] [INFO] [tso_client.go:155] ["tso client is closed"] [2024/05/06 19:55:43.323 +08:00] [INFO] [pd_service_discovery.go:637] ["[pd] close pd service discovery client"] + set +x table processor_delay.t26 exists table processor_delay.t27 exists table processor_delay.t28 not exists for 1-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc already exists) table processor_delay.t28 not exists for 2-th check, retry later [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7365853834131847599, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449576054648471555, "checkpoint_time": "2024-05-06 19:55:46.358", "error": null }' + echo '{ "upstream_id": 7365853834131847599, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449576054648471555, "checkpoint_time": "2024-05-06 19:55:46.358", "error": null }' { "upstream_id": 7365853834131847599, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449576054648471555, "checkpoint_time": "2024-05-06 19:55:46.358", "error": null } ++ echo '{' '"upstream_id":' 7365853834131847599, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449576054648471555, '"checkpoint_time":' '"2024-05-06' '19:55:46.358",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7365853834131847599, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449576054648471555, '"checkpoint_time":' '"2024-05-06' '19:55:46.358",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.14354.out cli unsafe delete-service-gc-safepoint Confirm that you know what this command will do and use it at your own risk [Y/N] CDC service GC safepoint truncated in PD! PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... table test.t2 not exists for 1-th check, retry later + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.14387.out cli unsafe reset --no-confirm --pd=http://127.0.0.1:2379 + start_tidb_cluster --workdir /tmp/tidb_cdc_test/sql_mode The 1 times to try to start tidb cluster... reset and all metadata truncated in PD! PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... table processor_delay.t28 not exists for 3-th check, retry later table test.t2 not exists for 2-th check, retry later + set +x ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2700 0 --:--:-- --:--:-- --:--:-- 2728 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-06 19:55:38.422","puller_resolved_ts":"2024-05-06 19:55:33.422","last_synced_ts":"2024-05-06 19:53:23.371","now_ts":"2024-05-06 19:55:41.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '19:55:38.422","puller_resolved_ts":"2024-05-06' '19:55:33.422","last_synced_ts":"2024-05-06' '19:53:23.371","now_ts":"2024-05-06' '19:55:41.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' + kill_pd ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo + info='jenkins 9546 8.8 0.0 13307732 146764 ? Sl 19:53 0:14 pd-server --advertise-client-urls http://127.0.0.1:2379 --client-urls http://0.0.0.0:2379 --advertise-peer-urls http://127.0.0.1:2380 --peer-urls http://0.0.0.0:2380 --config /tmp/tidb_cdc_test/synced_status_with_redo/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status_with_redo/pd1.log --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/pd1 --name=pd1 --initial-cluster=pd1=http://127.0.0.1:2380 jenkins 9613 6.2 0.0 13759632 139132 ? Sl 19:53 0:10 pd-server --advertise-client-urls http://127.0.0.1:2479 --client-urls http://0.0.0.0:2479 --advertise-peer-urls http://127.0.0.1:2480 --peer-urls http://0.0.0.0:2480 --config /tmp/tidb_cdc_test/synced_status_with_redo/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status_with_redo/down_pd.log --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/down_pd' ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo ++ awk '{print $2}' ++ xargs kill -9 + sleep 20 {"level":"warn","ts":1714996546.4834313,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0020cd340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714996546.483506,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-05-06T19:55:50.323393+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000a31340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:55:50.323721+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000c05340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:55:50.380991+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0008befc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} table processor_delay.t28 not exists for 4-th check, retry later table test.t2 not exists for 3-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/sql_mode Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... table processor_delay.t28 exists table processor_delay.t29 exists table processor_delay.t30 not exists for 1-th check, retry later table test.t2 not exists for 4-th check, retry later + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.14520.out cli unsafe resolve-lock --region=30 PASS \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } coverage: 2.2% of statements in github.com/pingcap/tiflow/... [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } {"level":"warn","ts":"2024-05-06T19:55:56.324286+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000a31340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:55:56.324898+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000c05340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:55:56.382453+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0008befc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} table processor_delay.t30 not exists for 2-th check, retry later [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.14557.out cli unsafe resolve-lock --region=30 --ts=449576056300240901 PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... table test.t2 not exists for 5-th check, retry later Verifying downstream PD is started... table processor_delay.t30 not exists for 3-th check, retry later + set +x % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 218 100 212 100 6 95840 2712 --:--:-- --:--:-- --:--:-- 103k { "error_msg": "[CDC:ErrAPIInvalidParam]invalid log level: json: cannot unmarshal string into Go value of type struct { Level string \"json:\\\"log_level\\\"\" }", "error_code": "CDC:ErrAPIInvalidParam" table test.t2 not exists for 6-th check, retry later Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table processor_delay.t30 not exists for 4-th check, retry later ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.t2 not exists for 7-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0{"level":"warn","ts":"2024-05-06T19:56:02.32489+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000a31340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:56:02.326598+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000c05340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:56:02.384095+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0008befc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} } % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 182 100 182 0 0 198k 0 --:--:-- --:--:-- --:--:-- 177k { "version": "v8.1.0-alpha-23-ge1a27adc6", "git_hash": "e1a27adc69acc9d50db0c3fea2945bbd6d983e30", "id": "fb2dbf2d-a2e4-4f67-8bef-bed9d21f6ccc", "pid": 13446, "is_owner": true }wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... table processor_delay.t30 exists table processor_delay.t31 not exists for 1-th check, retry later wait process cdc.test exit for 3-th time... table test.t2 not exists for 8-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:56:04 CST 2024] <<<<<< run test case cli_with_auth success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3708f4f40014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3, pid:9412, start at 2024-05-06 19:55:26.445668808 +0800 CST m=+5.258375381 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:57:26.452 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:55:26.446 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:45:26.446 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3708f4f40014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3, pid:9412, start at 2024-05-06 19:55:26.445668808 +0800 CST m=+5.258375381 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:57:26.452 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:55:26.446 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:45:26.446 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3708f6640017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3, pid:9485, start at 2024-05-06 19:55:26.536664071 +0800 CST m=+5.292337636 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:57:26.543 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:55:26.538 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:45:26.538 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'normal'@'%' start tidb cluster failed The 2 times to try to start tidb cluster... table processor_delay.t31 exists table processor_delay.t32 not exists for 1-th check, retry later table test.t2 not exists for 9-th check, retry later table processor_delay.t32 not exists for 2-th check, retry later table test.t2 not exists for 10-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0{"level":"warn","ts":"2024-05-06T19:56:08.326317+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000a31340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:56:08.328126+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000c05340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:56:08.385185+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0008befc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} table processor_delay.t32 not exists for 3-th check, retry later table test.t2 not exists for 11-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0{"level":"warn","ts":"2024-05-06T19:56:10.311454+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000c05340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2024-05-06T19:56:10.311517+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-05-06T19:56:10.312977+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000a31340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2024-05-06T19:56:10.31304+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-05-06T19:56:10.371676+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0008befc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":"2024-05-06T19:56:10.371734+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} table processor_delay.t32 not exists for 4-th check, retry later table test.t2 not exists for 12-th check, retry later table processor_delay.t32 exists table processor_delay.t33 not exists for 1-th check, retry later table test.t2 not exists for 13-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0{"level":"warn","ts":"2024-05-06T19:56:14.327267+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000a31340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:56:14.328936+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000c05340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:56:14.386232+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0008befc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} table processor_delay.t33 exists table processor_delay.t34 not exists for 1-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:14 --:--:-- 0{"level":"warn","ts":1714996575.761823,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002d0c000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714996575.7618868,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1714996575.8357227,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0038301c0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1714996575.8358076,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} table test.t2 not exists for 14-th check, retry later table processor_delay.t34 not exists for 2-th check, retry later table test.t2 exists check diff successfully wait process cdc.test exit for 1-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/savepoint/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/sql_mode Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... table processor_delay.t34 not exists for 3-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:56:20 CST 2024] <<<<<< run test case resolve_lock success! >>>>>> 0 0 0 0 0 0 0 0 --:--:-- 0:00:15 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:17 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:18 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0{"level":"warn","ts":"2024-05-06T19:56:20.329107+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000a31340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:56:20.32974+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000c05340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:56:20.388134+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0008befc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0{"level":"warn","ts":1714996581.4850557,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0020cd340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714996581.4851131,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} table processor_delay.t34 not exists for 4-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/savepoint Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... table processor_delay.t34 exists table processor_delay.t35 exists table processor_delay.t36 not exists for 1-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... table processor_delay.t36 not exists for 2-th check, retry later Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) 0 0 0 0 0 0 0 0 --:--:-- 0:00:21 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:22 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:23 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:24 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:25 --:--:-- 0{"level":"warn","ts":"2024-05-06T19:56:26.330453+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000a31340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:56:26.330925+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000c05340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:56:26.389095+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0008befc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table processor_delay.t36 not exists for 3-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3708f4f40014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3, pid:9412, start at 2024-05-06 19:55:26.445668808 +0800 CST m=+5.258375381 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:57:26.452 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:55:26.446 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:45:26.446 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3708f4f40014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3, pid:9412, start at 2024-05-06 19:55:26.445668808 +0800 CST m=+5.258375381 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:57:26.452 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:55:26.446 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:45:26.446 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3708f6640017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3, pid:9485, start at 2024-05-06 19:55:26.536664071 +0800 CST m=+5.292337636 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:57:26.543 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:55:26.538 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:45:26.538 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'normal'@'%' start tidb cluster failed The 3 times to try to start tidb cluster... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t36 not exists for 4-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) 0 0 0 0 0 0 0 0 --:--:-- 0:00:26 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:27 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:28 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 27 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 33 + synced_status='{ "error_msg": "[CDC:ErrPDEtcdAPIError]etcd api call error: context deadline exceeded", "error_code": "CDC:ErrPDEtcdAPIError" }' ++ echo '{' '"error_msg":' '"[CDC:ErrPDEtcdAPIError]etcd' api call error: context deadline 'exceeded",' '"error_code":' '"CDC:ErrPDEtcdAPIError"' '}' ++ jq -r .error_code + error_code=CDC:ErrPDEtcdAPIError + cleanup_process cdc.test wait process cdc.test exit for 1-th time... table processor_delay.t36 not exists for 5-th check, retry later wait process cdc.test exit for 2-th time... {"level":"warn","ts":"2024-05-06T19:56:32.331456+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000c05340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:56:32.332082+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000a31340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:56:32.390289+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0008befc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} wait process cdc.test exit for 3-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit + stop_tidb_cluster table processor_delay.t36 exists table processor_delay.t37 exists table processor_delay.t38 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370d0f140013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz, pid:15483, start at 2024-05-06 19:56:33.630927819 +0800 CST m=+5.232596388 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:58:33.637 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:56:33.605 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:46:33.605 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370d0f140013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz, pid:15483, start at 2024-05-06 19:56:33.630927819 +0800 CST m=+5.232596388 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:58:33.637 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:56:33.605 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:46:33.605 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370d108c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz, pid:15565, start at 2024-05-06 19:56:33.717722045 +0800 CST m=+5.263352942 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:58:33.728 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:56:33.699 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:46:33.699 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/savepoint/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/savepoint/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/move_table/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table processor_delay.t38 not exists for 2-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.cli.16908.out cli tso query --pd=http://127.0.0.1:2379 table processor_delay.t38 not exists for 3-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/move_table Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... + set +x + tso='449576067925278722 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576067925278722 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:56:38 CST 2024] <<<<<< START cdc server in savepoint case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.1694616948.out server --log-file /tmp/tidb_cdc_test/savepoint/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/savepoint/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 pass check, checkpoint tso not forward after 10s run task successfully wait process 8254 exit for 1-th time... wait process 8254 exit for 2-th time... wait process 8254 exit for 3-th time... wait process 8254 exit for 4-th time... /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (8254) - No such process wait process 8254 exit for 5-th time... process 8254 already exit [Mon May 6 19:56:33 CST 2024] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.89388940.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:56:36 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7365853838014817680 Namespace:default ID:ddl-only-block-related-table SinkURI:file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-10137?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:55:07.212378848 +0800 CST StartTs:449576044353552388 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003aaeab0 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449576044379766790} {CheckpointTs:449576049478467609 MinTableBarrierTs:449576067658153987 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449576049478467609, checkpointTs: 449576049478467609, state: Preparing span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449576049478467609, checkpointTs: 449576049478467609, state: Preparing span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449576049478467609, checkpointTs: 449576049478467609, state: Preparing span: {table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}, resolvedTs: 449576049478467609, checkpointTs: 449576049478467609, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9b1de08a-e510-4af4-9b99-74f83166bd42 {"id":"9b1de08a-e510-4af4-9b99-74f83166bd42","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996594} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1a82b20 9b1de08a-e510-4af4-9b99-74f83166bd42 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7365853838014817680,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-10137?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:55:07.212378848+08:00","start-ts":449576044353552388,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449576044379766790} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":449576049478467609,"min-table-barrier-ts":449576067658153987,"admin-job-type":0} /tidb/cdc/default/default/task/position/9b1de08a-e510-4af4-9b99-74f83166bd42/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365853838014817680 {"id":7365853838014817680,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7365853838014817680 Namespace:default ID:ddl-only-block-related-table SinkURI:file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-10137?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:55:07.212378848 +0800 CST StartTs:449576044353552388 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003aaeab0 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449576044379766790} {CheckpointTs:449576049478467609 MinTableBarrierTs:449576067658153987 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449576049478467609, checkpointTs: 449576049478467609, state: Preparing span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449576049478467609, checkpointTs: 449576049478467609, state: Preparing span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449576049478467609, checkpointTs: 449576049478467609, state: Preparing span: {table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}, resolvedTs: 449576049478467609, checkpointTs: 449576049478467609, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9b1de08a-e510-4af4-9b99-74f83166bd42 {"id":"9b1de08a-e510-4af4-9b99-74f83166bd42","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996594} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1a82b20 9b1de08a-e510-4af4-9b99-74f83166bd42 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7365853838014817680,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-10137?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:55:07.212378848+08:00","start-ts":449576044353552388,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":18000000+ grep -q 'failed to get info:' 00000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449576044379766790} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":449576049478467609,"min-table-barrier-ts":449576067658153987,"admin-job-type":0} /tidb/cdc/default/default/task/position/9b1de08a-e510-4af4-9b99-74f83166bd42/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365853838014817680 {"id":7365853838014817680,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7365853838014817680 Namespace:default ID:ddl-only-block-related-table SinkURI:file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-10137?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:55:07.212378848 +0800 CST StartTs:449576044353552388 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003aaeab0 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449576044379766790} {CheckpointTs:449576049478467609 MinTableBarrierTs:449576067658153987 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449576049478467609, checkpointTs: 449576049478467609, state: Preparing span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449576049478467609, checkpointTs: 449576049478467609, state: Preparing span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449576049478467609, checkpointTs: 449576049478467609, state: Preparing span: {table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}, resolvedTs: 449576049478467609, checkpointTs: 449576049478467609, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9b1de08a-e510-4af4-9b99-74f83166bd42 {"id":"9b1de08a-e510-4af4-9b99-74f83166bd42","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996594} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1a82b20 9b1de08a-e510-4af4-9b99-74f83166bd42 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7365853838014817680,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-10137?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:55:07.212378848+08:00","start-ts":449576044353552388,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449576044379766790} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":449576049478467609,"min-table-barrier-ts":449576067658153987,"admin-job-type":0} /tidb/cdc/default/default/task/position/9b1de08a-e510-4af4-9b99-74f83166bd42/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365853838014817680 + grep -q 'etcd info' {"id":7365853838014817680,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x check diff failed 1-th time, retry later check diff failed 2-th time, retry later table processor_delay.t38 not exists for 4-th check, retry later Verifying downstream PD is started... check diff successfully check_ts_forward ddl-only-block-related-table table processor_delay.t38 not exists for 5-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:56:41 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6e32eae9-cfa0-4c4b-8f12-f468392668af {"id":"6e32eae9-cfa0-4c4b-8f12-f468392668af","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996598} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc3198f33 6e32eae9-cfa0-4c4b-8f12-f468392668af /tidb/cdc/default/default/upstream/7365854248623427169 {"id":7365854248623427169,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6e32eae9-cfa0-4c4b-8f12-f468392668af {"id":"6e32eae9-cfa0-4c4b-8f12-f468392668af","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996598} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc3198f33 6e32eae9-cfa0-4c4b-8f12-f468392668af /tidb/cdc/default/default/upstream/7365854248623427169 {"id":7365854248623427169,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6e32eae9-cfa0-4c4b-8f12-f468392668af {"id":"6e32eae9-cfa0-4c4b-8f12-f468392668af","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996598} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc3198f33 6e32eae9-cfa0-4c4b-8f12-f468392668af /tidb/cdc/default/default/upstream/7365854248623427169 {"id":7365854248623427169,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.cli.17008.out cli changefeed create --start-ts=449576067925278722 '--sink-uri=file:///tmp/tidb_cdc_test/savepoint/storage_test/ticdc-savepoint-test-13124?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: e3fc78a8-e020-4da0-9548-02c497f20e34 Info: {"upstream_id":7365854248623427169,"namespace":"default","id":"e3fc78a8-e020-4da0-9548-02c497f20e34","sink_uri":"file:///tmp/tidb_cdc_test/savepoint/storage_test/ticdc-savepoint-test-13124?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:56:41.979908681+08:00","start_ts":449576067925278722,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576067925278722,"checkpoint_ts":449576067925278722,"checkpoint_time":"2024-05-06 19:56:37.005"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release start tidb cluster in /tmp/tidb_cdc_test/sql_mode Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... changefeed is working normally rts: 449576068968874000->449576069231018000 checkpoint: 449576068968874000->449576069231018000 run task successfully + set +x + workdir=/tmp/tidb_cdc_test/savepoint + sink_uri='file:///tmp/tidb_cdc_test/savepoint/storage_test/ticdc-savepoint-test-13124?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/savepoint ++ date + echo '[Mon May 6 19:56:43 CST 2024] <<<<<< START storage consumer in savepoint case >>>>>>' [Mon May 6 19:56:43 CST 2024] <<<<<< START storage consumer in savepoint case >>>>>> + cd /tmp/tidb_cdc_test/savepoint + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/savepoint + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/savepoint/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/savepoint/storage_test/ticdc-savepoint-test-13124?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table processor_delay.t38 exists table processor_delay.t39 exists table processor_delay.t40 not exists for 1-th check, retry later table savepoint.finish_mark not exists for 1-th check, retry later wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:56:45 CST 2024] <<<<<< run test case ddl_only_block_related_table success! >>>>>> table processor_delay.t40 not exists for 2-th check, retry later table savepoint.finish_mark not exists for 2-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t40 not exists for 3-th check, retry later + run_case_with_unavailable_tikv conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status_with_redo + mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory table savepoint.finish_mark not exists for 3-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table processor_delay.t40 not exists for 4-th check, retry later table savepoint.finish_mark not exists for 4-th check, retry later chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t40 not exists for 5-th check, retry later table savepoint.finish_mark not exists for 5-th check, retry later *************************** 1. row *************************** count(distinct region_id): 1 split_and_random_merge scale: 80 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3708f4f40014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3, pid:9412, start at 2024-05-06 19:55:26.445668808 +0800 CST m=+5.258375381 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:57:26.452 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:55:26.446 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:45:26.446 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3708f4f40014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3, pid:9412, start at 2024-05-06 19:55:26.445668808 +0800 CST m=+5.258375381 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:57:26.452 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:55:26.446 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:45:26.446 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3708f6640017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-175s3, pid:9485, start at 2024-05-06 19:55:26.536664071 +0800 CST m=+5.292337636 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:57:26.543 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:55:26.538 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:45:26.538 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'normal'@'%' start tidb cluster failed + run_sql 'set global sql_mode='\''ANSI_QUOTES'\'';' 127.0.0.1 4000 + run_sql 'set global sql_mode='\''ANSI_QUOTES'\'';' 127.0.0.1 3306 ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.cli.14716.out cli tso query --pd=http://127.0.0.1:2379 table savepoint.finish_mark not exists for 6-th check, retry later table processor_delay.t40 exists table processor_delay.t41 exists table processor_delay.t42 not exists for 1-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + tso='449576072238333955 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576072238333955 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449576072238333955 + run_cdc_server --workdir /tmp/tidb_cdc_test/sql_mode --binary cdc.test [Mon May 6 19:56:55 CST 2024] <<<<<< START cdc server in sql_mode case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.1475114753.out server --log-file /tmp/tidb_cdc_test/sql_mode/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sql_mode/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t42 not exists for 2-th check, retry later table savepoint.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370e56340004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-pst3l, pid:15899, start at 2024-05-06 19:56:54.56066232 +0800 CST m=+10.124515559 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:58:54.599 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:56:54.642 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:46:54.642 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370e56340004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-pst3l, pid:15899, start at 2024-05-06 19:56:54.56066232 +0800 CST m=+10.124515559 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:58:54.599 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:56:54.642 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:46:54.642 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370e5590000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-pst3l, pid:15932, start at 2024-05-06 19:56:54.539040277 +0800 CST m=+9.995193583 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:58:54.589 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:56:54.600 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:46:54.600 +0800 All versions after safe point can be accessed. (DO NOT EDIT) wait process cdc.test exit for 2-th time... Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/move_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/move_table/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/move_table/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/move_table/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/move_table/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Mon May 6 19:56:57 CST 2024] <<<<<< run test case savepoint success! >>>>>> table processor_delay.t42 not exists for 3-th check, retry later *************************** 1. row *************************** count(distinct region_id): 18 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:56:58 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/test-1 {UpstreamID:7365853961037049289 Namespace:default ID:test-1 SinkURI:mysql://root@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-06 19:55:36.88993632 +0800 CST StartTs:449576050844499971 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0016d7170 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449576052154957831} {CheckpointTs:449576052522221571 MinTableBarrierTs:449576052522221571 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2c730f9d-d7fb-4209-9d0d-b99184cd4c96 {"id":"2c730f9d-d7fb-4209-9d0d-b99184cd4c96","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996533} /tidb/cdc/default/__cdc_meta__/capture/db00c8f3-573e-4a1a-b814-15b8a092f1ad {"id":"db00c8f3-573e-4a1a-b814-15b8a092f1ad","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996615} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc21ad73f 2c730f9d-d7fb-4209-9d0d-b99184cd4c96 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc367afd0 db00c8f3-573e-4a1a-b814-15b8a092f1ad /tidb/cdc/default/default/changefeed/info/test-1 {"upstream-id":7365853961037049289,"namespace":"default","changefeed-id":"test-1","sink-uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-06T19:55:36.88993632+08:00","start-ts":449576050844499971,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449576052154957831} /tidb/cdc/default/default/changefeed/status/test-1 {"checkpoint-ts":449576052522221571,"min-table-barrier-ts":449576052522221571,"admin-job-type":0} /tidb/cdc/default/default/task/position/2c730f9d-d7fb-4209-9d0d-b99184cd4c96/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/db00c8f3-573e-4a1a-b814-15b8a092f1ad/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365853961037049289 {"id":7365853961037049289,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/test-1 {UpstreamID:7365853961037049289 Namespace:default ID:test-1 SinkURI:mysql://root@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-06 19:55:36.88993632 +0800 CST StartTs:449576050844499971 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0016d7170 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449576052154957831} {CheckpointTs:449576052522221571 MinTableBarrierTs:449576052522221571 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2c730f9d-d7fb-4209-9d0d-b99184cd4c96 {"id":"2c730f9d-d7fb-4209-9d0d-b99184cd4c96","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996533} /tidb/cdc/default/__cdc_meta__/capture/db00c8f3-573e-4a1a-b814-15b8a092f1ad {"id":"db00c8f3-573e-4a1a-b814-15b8a092f1ad","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996615} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc21ad73f 2c730f9d-d7fb-4209-9d0d-b99184cd4c96 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc367afd0 db00c8f3-573e-4a1a-b814-15b8a092f1ad /tidb/cdc/default/default/changefeed/info/test-1 {"upstream-id":7365853961037049289,"namespace":"default","changefeed-id":"test-1","sink-uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-06T19:55:36.88993632+08:00","start-ts":449576050844499971,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449576052154957831} /tidb/cdc/default/default/changefeed/status/test-1 {"checkpoint-ts":449576052522221571,"min-table-barrier-ts":449576052522221571,"admin-job-type":0} /tidb/cdc/default/default/task/position/2c730f9d-d7fb-4209-9d0d-b99184cd4c96/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/db00c8f3-573e-4a1a-b814-15b8a092f1ad/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365853961037049289 {"id":7365853961037049289,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/test-1 {UpstreamID:7365853961037049289 Namespace:default ID:test-1 SinkURI:mysql://root@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-06 19:55:36.88993632 +0800 CST StartTs:449576050844499971 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0016d7170 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449576052154957831} {CheckpointTs:449576052522221571 MinTableBarrierTs:449576052522221571 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2c730f9d-d7fb-4209-9d0d-b99184cd4c96 {"id":"2c730f9d-d7fb-4209-9d0d-b99184cd4c96","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996533} /tidb/cdc/default/__cdc_meta__/capture/db00c8f3-573e-4a1a-b814-15b8a092f1ad {"id":"db00c8f3-573e-4a1a-b814-15b8a092f1ad","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996615} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc21ad73f 2c730f9d-d7fb-4209-9d0d-b99184cd4c96 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc367afd0 db00c8f3-573e-4a1a-b814-15b8a092f1ad /tidb/cdc/default/default/changefeed/info/test-1 {"upstream-id":7365853961037049289,"namespace":"default","changefeed-id":"test-1","sink-uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-06T19:55:36.88993632+08:00","start-ts":449576050844499971,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449576052154957831} /tidb/cdc/default/default/changefeed/status/test-1 {"checkpoint-ts":449576052522221571,"min-table-barrier-ts":449576052522221571,"admin-job-type":0} /tidb/cdc/default/default/task/position/2c730f9d-d7fb-4209-9d0d-b99184cd4c96/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/db00c8f3-573e-4a1a-b814-15b8a092f1ad/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"+ grep -q 'etcd info' error":null,"warning":null} /tidb/cdc/default/default/upstream/7365853961037049289 {"id":7365853961037049289,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449576072238333955 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-2 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.cli.14807.out cli changefeed create --start-ts=449576072238333955 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-2 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.cli.17164.out cli tso query --pd=http://127.0.0.1:2379 table processor_delay.t42 not exists for 4-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/changefeed_auto_stop/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Create changefeed successfully! ID: test-2 Info: {"upstream_id":7365853961037049289,"namespace":"default","id":"test-2","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-06T19:57:00.912369383+08:00","start_ts":449576072238333955,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576072238333955,"checkpoint_ts":449576072238333955,"checkpoint_time":"2024-05-06 19:56:53.458"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... + set +x + tso='449576073859432450 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576073859432450 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x table processor_delay.t42 not exists for 5-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ***************** properties ***************** "updateproportion"="0" "requestdistribution"="uniform" "readproportion"="0" "threadcount"="10" "insertproportion"="0" "mysql.user"="root" "workload"="core" "mysql.port"="4000" "readallfields"="true" "scanproportion"="0" "mysql.host"="127.0.0.1" "mysql.db"="move_table" "recordcount"="10000" "operationcount"="0" "dotransactions"="false" ********************************************** table region_merge.t1 exists check diff failed 1-th time, retry later + set +x + run_sql 'use test; create table t2(id bigint primary key, a date); insert into t2 values(1, '\''2023-02-08'\'');' 127.0.0.1 4000 table processor_delay.t42 exists table processor_delay.t43 exists table processor_delay.t44 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370ec6dc0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l, pid:12879, start at 2024-05-06 19:57:01.770489822 +0800 CST m=+5.461390803 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:59:01.777 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:57:01.751 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:47:01.751 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370ec6dc0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l, pid:12879, start at 2024-05-06 19:57:01.770489822 +0800 CST m=+5.461390803 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:59:01.777 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:57:01.751 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:47:01.751 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370ec6d40013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l, pid:12948, start at 2024-05-06 19:57:01.774958364 +0800 CST m=+5.412992269 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:59:01.785 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:57:01.749 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:47:01.749 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 2-th time, retry later table processor_delay.t44 not exists for 2-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/changefeed_auto_stop Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... + cd /tmp/tidb_cdc_test/synced_status_with_redo ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.14262.out cli tso query --pd=http://127.0.0.1:2379 check diff failed 3-th time, retry later table processor_delay.t44 not exists for 3-th check, retry later + set +x + tso='449576075775442945 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576075775442945 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449576075775442945 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test [Mon May 6 19:57:08 CST 2024] <<<<<< START cdc server in synced_status_with_redo case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.1430114303.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 4-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/run.sh using Sink-Type: storage... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status + source /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/synced_status + CDC_BINARY=cdc.test + SINK_TYPE=storage + CDC_COUNT=3 + DB_COUNT=4 + trap stop_tidb_cluster EXIT + run_normal_case_and_unavailable_pd conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status The 1 times to try to start tidb cluster... table processor_delay.t44 not exists for 4-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff failed 5-th time, retry later table processor_delay.t44 not exists for 5-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:57:11 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/70950bc8-ad53-42e5-95c6-8160c3716035 {"id":"70950bc8-ad53-42e5-95c6-8160c3716035","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996628} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc384943a 70950bc8-ad53-42e5-95c6-8160c3716035 /tidb/cdc/default/default/upstream/7365854368270930065 {"id":7365854368270930065,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/70950bc8-ad53-42e5-95c6-8160c3716035 {"id":"70950bc8-ad53-42e5-95c6-8160c3716035","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996628} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc384943a 70950bc8-ad53-42e5-95c6-8160c3716035 /tidb/cdc/default/default/upstream/7365854368270930065 {"id":7365854368270930065,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/70950bc8-ad53-42e5-95c6-8160c3716035 {"id":"70950bc8-ad53-42e5-95c6-8160c3716035","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996628} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc384943a 70950bc8-ad53-42e5-95c6-8160c3716035 /tidb/cdc/default/default/upstream/7365854368270930065 {"id":7365854368270930065,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449576075775442945 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.14360.out cli changefeed create --start-ts=449576075775442945 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7365854368270930065,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-06T19:57:11.973877738+08:00","start_ts":449576075775442945,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576075775442945,"checkpoint_ts":449576075775442945,"checkpoint_time":"2024-05-06 19:57:06.951"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... INSERT - Takes(s): 10.0, Count: 9955, OPS: 996.5, Avg(us): 9946, Min(us): 1508, Max(us): 68428, 95th(us): 24000, 99th(us): 33000 Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Run finished, takes 10.164712219s INSERT - Takes(s): 10.2, Count: 10000, OPS: 984.8, Avg(us): 9961, Min(us): 1508, Max(us): 68428, 95th(us): 24000, 99th(us): 33000 [Mon May 6 19:57:12 CST 2024] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.1723917241.out server --log-file /tmp/tidb_cdc_test/move_table/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data1 --cluster-id default --addr 127.0.0.1:8300 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 6-th time, retry later + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' table processor_delay.t44 exists table processor_delay.t45 exists table processor_delay.t46 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... check diff successfully wait process cdc.test exit for 1-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:57:15 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b92fe9df-19db-4d86-9166-94f2b1f56ac8 {"id":"b92fe9df-19db-4d86-9166-94f2b1f56ac8","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996633} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc355a795 b92fe9df-19db-4d86-9166-94f2b1f56ac8 /tidb/cdc/default/default/upstream/7365854308017255117 {"id":7365854308017255117,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b92fe9df-19db-4d86-9166-94f2b1f56ac8 {"id":"b92fe9df-19db-4d86-9166-94f2b1f56ac8","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996633} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc355a795 b92fe9df-19db-4d86-9166-94f2b1f56ac8 /tidb/cdc/default/default/upstream/7365854308017255117 {"id":7365854308017255117,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b92fe9df-19db-4d86-9166-94f2b1f56ac8 {"id":"b92fe9df-19db-4d86-9166-94f2b1f56ac8","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996633} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc355a795 b92fe9df-19db-4d86-9166-94f2b1f56ac8 /tidb/cdc/default/default/upstream/7365854308017255117 {"id":7365854308017255117,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.cli.17275.out cli changefeed create --start-ts=449576073859432450 '--sink-uri=file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-18127?protocol=canal-json&enable-tidb-extension=true' wait process cdc.test exit for 2-th time... table processor_delay.t46 not exists for 2-th check, retry later Verifying downstream PD is started... wait process cdc.test exit for 3-th time... Create changefeed successfully! ID: 6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d Info: {"upstream_id":7365854308017255117,"namespace":"default","id":"6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d","sink_uri":"file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-18127?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:57:16.504481889+08:00","start_ts":449576073859432450,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576073859432450,"checkpoint_ts":449576073859432450,"checkpoint_time":"2024-05-06 19:56:59.642"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:57:17 CST 2024] <<<<<< run test case region_merge success! >>>>>> table test.t1 exists + sleep 5 table processor_delay.t46 not exists for 3-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/move_table + sink_uri='file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-18127?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/move_table ++ date + echo '[Mon May 6 19:57:18 CST 2024] <<<<<< START storage consumer in move_table case >>>>>>' [Mon May 6 19:57:18 CST 2024] <<<<<< START storage consumer in move_table case >>>>>> + cd /tmp/tidb_cdc_test/move_table + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/move_table + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/move_table/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-18127?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' [Mon May 6 19:57:18 CST 2024] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS= + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.1730617308.out server --log-file /tmp/tidb_cdc_test/move_table/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data2 --cluster-id default --addr 127.0.0.1:8301 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo '' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370fbfc40015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-nrrkt, pid:10148, start at 2024-05-06 19:57:17.717673796 +0800 CST m=+5.189987704 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:59:17.726 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:57:17.731 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:47:17.731 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370fbfc40015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-nrrkt, pid:10148, start at 2024-05-06 19:57:17.717673796 +0800 CST m=+5.189987704 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:59:17.726 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:57:17.731 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:47:17.731 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370fc2d0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-nrrkt, pid:10219, start at 2024-05-06 19:57:17.886657932 +0800 CST m=+5.309570085 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:59:17.896 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:57:17.876 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:47:17.876 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table processor_delay.t46 not exists for 4-th check, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t46 not exists for 5-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:57:21 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {UpstreamID:7365854308017255117 Namespace:default ID:6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d SinkURI:file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-18127?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:57:16.504481889 +0800 CST StartTs:449576073859432450 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037b1a70 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449576078276296706} {CheckpointTs:449576074475470861 MinTableBarrierTs:449576079377301509 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/03134e50-ddf5-4cfb-9d25-792d52f5d3dd {"id":"03134e50-ddf5-4cfb-9d25-792d52f5d3dd","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996639} /tidb/cdc/default/__cdc_meta__/capture/b92fe9df-19db-4d86-9166-94f2b1f56ac8 {"id":"b92fe9df-19db-4d86-9166-94f2b1f56ac8","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996633} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc355a795 b92fe9df-19db-4d86-9166-94f2b1f56ac8 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc355a7f4 03134e50-ddf5-4cfb-9d25-792d52f5d3dd /tidb/cdc/default/default/changefeed/info/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"upstream-id":7365854308017255117,"namespace":"default","changefeed-id":"6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d","sink-uri":"file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-18127?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:57:16.504481889+08:00","start-ts":449576073859432450,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449576078276296706} /tidb/cdc/default/default/changefeed/status/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"checkpoint-ts":449576074475470861,"min-table-barrier-ts":449576079377301509,"admin-job-type":0} /tidb/cdc/default/default/task/position/03134e50-ddf5-4cfb-9d25-792d52f5d3dd/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b92fe9df-19db-4d86-9166-94f2b1f56ac8/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365854308017255117 {"id":7365854308017255117,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {UpstreamID:7365854308017255117 Namespace:default ID:6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d SinkURI:file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-18127?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:57:16.504481889 +0800 CST StartTs:449576073859432450 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037b1a70 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449576078276296706} {CheckpointTs:449576074475470861 MinTableBarrierTs:449576079377301509 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/03134e50-ddf5-4cfb-9d25-792d52f5d3dd {"id":"03134e50-ddf5-4cfb-9d25-792d52f5d3dd","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996639} /tidb/cdc/default/__cdc_meta__/capture/b92fe9df-19db-4d86-9166-94f2b1f56ac8 {"id":"b92fe9df-19db-4d86-9166-94f2b1f56ac8","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996633} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc355a795 b92fe9df-19db-4d86-9166-94f2b1f56ac8 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc355a7f4 03134e50-ddf5-4cfb-9d25-792d52f5d3dd /tidb/cdc/default/default/changefeed/info/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"upstream-id":7365854308017255117,"namespace":"default","changefeed-id":"6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d","sink-uri":"file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-18127?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:57:16.504481889+08:00","start-ts":449576073859432450,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449576078276296706} /tidb/cdc/default/default/changefeed/status/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"checkpoint-ts":449576074475470861,"min-table-barrier-ts":449576079377301509,"admin-job-type":0} /tidb/cdc/default/default/task/position/03134e50-ddf5-4cfb-9d25-792d52f5d3dd/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b92fe9df-19db-4d86-9166-94f2b1f56ac8/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365854308017255117 {"id":7365854308017255117,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {UpstreamID:7365854308017255117 Namespace:default ID:6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d SinkURI:file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-18127?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:57:16.504481889 +0800 CST StartTs:449576073859432450 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037b1a70 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449576078276296706} {CheckpointTs:449576074475470861 MinTableBarrierTs:449576079377301509 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/03134e50-ddf5-4cfb-9d25-792d52f5d3dd {"id":"03134e50-ddf5-4cfb-9d25-792d52f5d3dd","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996639} /tidb/cdc/default/__cdc_meta__/capture/b92fe9df-19db-4d86-9166-94f2b1f56ac8 {"id":"b92fe9df-19db-4d86-9166-94f2b1f56ac8","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996633} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc355a795 b92fe9df-19db-4d86-9166-94f2b1f56ac8 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc355a7f4 03134e50-ddf5-4cfb-9d25-792d52f5d3dd /tidb/cdc/default/default/changefeed/info/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"upstream-id":7365854308017255117,"namespace":"default","changefeed-id":"6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d","sink-uri":"file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-18127?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:57:16.504481889+08:00","start-ts":449576073859432450,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449576078276296706} /tidb/cdc/default/default/changefeed/status/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"checkpoint-ts":449576074475470861,"min-table-barrier-ts":449576079377301509,"admin-job-type":0} /tidb/cdc/default/default/task/position/03134e50-ddf5-4cfb-9d25-792d52f5d3dd/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b92fe9df-19db-4d86-9166-94f2b1f56ac8/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365854308017255117 {"id":7365854308017255117,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x [Mon May 6 19:57:21 CST 2024] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.1737717379.out server --log-file /tmp/tidb_cdc_test/move_table/cdc3.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data3 --cluster-id default --addr 127.0.0.1:8302 ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ***************** properties ***************** "workload"="core" "mysql.port"="4000" "mysql.user"="root" "mysql.db"="changefeed_auto_stop_1" "threadcount"="4" "mysql.host"="127.0.0.1" "dotransactions"="false" "requestdistribution"="uniform" "readproportion"="0" "insertproportion"="0" "scanproportion"="0" "recordcount"="20" "readallfields"="true" "updateproportion"="0" "operationcount"="0" ********************************************** Run finished, takes 9.053458ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3947.7, Avg(us): 1716, Min(us): 1059, Max(us): 3985, 95th(us): 4000, 99th(us): 4000 + kill_tikv ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo + info='jenkins 12302 18.5 0.4 3962336 1866188 ? Sl 19:56 0:05 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20160 --status-addr 127.0.0.1:20181 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tikv1.log --log-level debug -C /tmp/tidb_cdc_test/synced_status_with_redo/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status_with_redo/tikv1 jenkins 12303 18.8 0.4 3965924 1882364 ? Sl 19:56 0:05 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20161 --status-addr 127.0.0.1:20182 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tikv2.log --log-level debug -C /tmp/tidb_cdc_test/synced_status_with_redo/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status_with_redo/tikv2 jenkins 12304 26.0 0.4 3997152 1929248 ? Sl 19:56 0:07 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20162 --status-addr 127.0.0.1:20183 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tikv3.log --log-level debug -C /tmp/tidb_cdc_test/synced_status_with_redo/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status_with_redo/tikv3 jenkins 12306 25.2 0.4 3989476 1920304 ? Sl 19:56 0:07 tikv-server --pd 127.0.0.1:2479 -A 127.0.0.1:21160 --status-addr 127.0.0.1:21180 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tikv_down.log --log-level debug -C /tmp/tidb_cdc_test/synced_status_with_redo/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status_with_redo/tikv_down' ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 3476 0 --:--:-- --:--:-- --:--:-- 3521 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-06 19:57:19.352","puller_resolved_ts":"2024-05-06 19:57:13.452","last_synced_ts":"2024-05-06 19:57:13.951","now_ts":"2024-05-06 19:57:21.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '19:57:19.352","puller_resolved_ts":"2024-05-06' '19:57:13.452","last_synced_ts":"2024-05-06' '19:57:13.951","now_ts":"2024-05-06' '19:57:21.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '19:57:19.352","puller_resolved_ts":"2024-05-06' '19:57:13.452","last_synced_ts":"2024-05-06' '19:57:13.951","now_ts":"2024-05-06' '19:57:21.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ***************** properties ***************** "recordcount"="20" "mysql.user"="root" "mysql.db"="changefeed_auto_stop_2" "scanproportion"="0" "workload"="core" "mysql.port"="4000" "readproportion"="0" "readallfields"="true" "dotransactions"="false" "threadcount"="4" "updateproportion"="0" "insertproportion"="0" "operationcount"="0" "mysql.host"="127.0.0.1" "requestdistribution"="uniform" ********************************************** Run finished, takes 10.171494ms INSERT - Takes(s): 0.0, Count: 19, OPS: 3211.5, Avg(us): 1797, Min(us): 1071, Max(us): 4158, 95th(us): 5000, 99th(us): 5000 table processor_delay.t46 exists table processor_delay.t47 exists table processor_delay.t48 not exists for 1-th check, retry later ***************** properties ***************** "requestdistribution"="uniform" "scanproportion"="0" "updateproportion"="0" "threadcount"="4" "mysql.host"="127.0.0.1" "insertproportion"="0" "readallfields"="true" "readproportion"="0" "mysql.port"="4000" "dotransactions"="false" "workload"="core" "recordcount"="20" "mysql.user"="root" "operationcount"="0" "mysql.db"="changefeed_auto_stop_3" ********************************************** Run finished, takes 9.592537ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3678.0, Avg(us): 1740, Min(us): 989, Max(us): 4082, 95th(us): 5000, 99th(us): 5000 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:57:24 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {UpstreamID:7365854308017255117 Namespace:default ID:6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d SinkURI:file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-18127?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:57:16.504481889 +0800 CST StartTs:449576073859432450 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003d48e10 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449576078276296706} {CheckpointTs:449576075602427914 MinTableBarrierTs:449576080165306382 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/03134e50-ddf5-4cfb-9d25-792d52f5d3dd {"id":"03134e50-ddf5-4cfb-9d25-792d52f5d3dd","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996639} /tidb/cdc/default/__cdc_meta__/capture/761a6499-cd6d-4a2c-afd1-7fea1e2d348f {"id":"761a6499-cd6d-4a2c-afd1-7fea1e2d348f","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996642} /tidb/cdc/default/__cdc_meta__/capture/b92fe9df-19db-4d86-9166-94f2b1f56ac8 {"id":"b92fe9df-19db-4d86-9166-94f2b1f56ac8","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996633} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc355a795 b92fe9df-19db-4d86-9166-94f2b1f56ac8 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc355a7f4 03134e50-ddf5-4cfb-9d25-792d52f5d3dd /tidb/cdc/default/__cdc_meta__/owner/22318f4dc355a81d 761a6499-cd6d-4a2c-afd1-7fea1e2d348f /tidb/cdc/default/default/changefeed/info/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"upstream-id":7365854308017255117,"namespace":"default","changefeed-id":"6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d","sink-uri":"file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-18127?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:57:16.504481889+08:00","start-ts":449576073859432450,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449576078276296706} /tidb/cdc/default/default/changefeed/status/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"checkpoint-ts":449576075602427914,"min-table-barrier-ts":449576080165306382,"admin-job-type":0} /tidb/cdc/default/default/task/position/03134e50-ddf5-4cfb-9d25-792d52f5d3dd/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/761a6499-cd6d-4a2c-afd1-7fea1e2d348f/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b92fe9df-19db-4d86-9166-94f2b1f56ac8/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365854308017255117 {"id":7365854308017255117,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {UpstreamID:7365854308017255117 Namespace:default ID:6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d SinkURI:file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-18127?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:57:16.504481889 +0800 CST StartTs:449576073859432450 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003d48e10 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449576078276296706} {CheckpointTs:449576075602427914 MinTableBarrierTs:449576080165306382 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/03134e50-ddf5-4cfb-9d25-792d52f5d3dd {"id":"03134e50-ddf5-4cfb-9d25-792d52f5d3dd","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996639} /tidb/cdc/default/__cdc_meta__/capture/761a6499-cd6d-4a2c-afd1-7fea1e2d348f {"id":"761a6499-cd6d-4a2c-afd1-7fea1e2d348f","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996642} /tidb/cdc/default/__cdc_meta__/capture/b92fe9df-19db-4d86-9166-94f2b1f56ac8 {"id":"b92fe9df-19db-4d86-9166-94f2b1f56ac8","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996633} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc355a795 b92fe9df-19db-4d86-9166-94f2b1f56ac8 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc355a7f4 03134e50-ddf5-4cfb-9d25-792d52f5d3dd /tidb/cdc/default/__cdc_meta__/owner/22318f4dc355a81d 761a6499-cd6d-4a2c-afd1-7fea1e2d348f /tidb/cdc/default/default/changefeed/info/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"upstream-id":7365854308017255117,"namespace":"default","changefeed-id":"6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d","sink-uri":"file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-18127?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:57:16.504481889+08:00","start-ts":449576073859432450,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449576078276296706} /tidb/cdc/default/default/changefeed/status/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"checkpoint-ts":449576075602427914,"min-table-barrier-ts":449576080165306382,"admin-job-type":0} /tidb/cdc/default/default/task/position/03134e50-ddf5-4cfb-9d25-792d52f5d3dd/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/761a6499-cd6d-4a2c-afd1-7fea1e2d348f/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b92fe9df-19db-4d86-9166-94f2b1f56ac8/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365854308017255117 {"id":7365854308017255117,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {UpstreamID:7365854308017255117 Namespace:default ID:6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d SinkURI:file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-18127?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:57:16.504481889 +0800 CST StartTs:449576073859432450 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003d48e10 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449576078276296706} {CheckpointTs:449576075602427914 MinTableBarrierTs:449576080165306382 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/03134e50-ddf5-4cfb-9d25-792d52f5d3dd {"id":"03134e50-ddf5-4cfb-9d25-792d52f5d3dd","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996639} /tidb/cdc/default/__cdc_meta__/capture/761a6499-cd6d-4a2c-afd1-7fea1e2d348f {"id":"761a6499-cd6d-4a2c-afd1-7fea1e2d348f","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996642} /tidb/cdc/default/__cdc_meta__/capture/b92fe9df-19db-4d86-9166-94f2b1f56ac8 {"id":"b92fe9df-19db-4d86-9166-94f2b1f56ac8","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996633} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc355a795 b92fe9df-19db-4d86-9166-94f2b1f56ac8 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc355a7f4 03134e50-ddf5-4cfb-9d25-792d52f5d3dd /tidb/cdc/default/__cdc_meta__/owner/22318f4dc355a81d 761a6499-cd6d-4a2c-afd1-7fea1e2d348f /tidb/cdc/default/default/changefeed/info/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"upstream-id":7365854308017255117,"namespace":"default","changefeed-id":"6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d","sink-uri":"file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-18127?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:57:16.504481889+08:00","start-ts":449576073859432450,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449576078276296706} /tidb/cdc/default/default/changefeed/status/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"checkpoint-ts":449576075602427914,"min-table-barrier-ts":449576080165306382,"admin-job-type":0} /tidb/cdc/default/default/task/position/03134e50-ddf5-4cfb-9d25-792d52f5d3dd/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/761a6499-cd6d-4a2c-afd1-7fea1e2d348f/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b92fe9df-19db-4d86-9166-94f2b1f56ac8/6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365854308017255117 {"id":7365854308017255117,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table move_table.usertable not exists for 1-th check, retry later ***************** properties ***************** "threadcount"="4" "mysql.db"="changefeed_auto_stop_4" "mysql.port"="4000" "insertproportion"="0" "recordcount"="20" "mysql.host"="127.0.0.1" "requestdistribution"="uniform" "readallfields"="true" "operationcount"="0" "updateproportion"="0" "scanproportion"="0" "readproportion"="0" "dotransactions"="false" "mysql.user"="root" "workload"="core" ********************************************** Run finished, takes 9.956881ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3840.6, Avg(us): 1887, Min(us): 1031, Max(us): 4723, 95th(us): 5000, 99th(us): 5000 [Mon May 6 19:57:25 CST 2024] <<<<<< START cdc server in changefeed_auto_stop case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_auto_stop.1173111733.out server --log-file /tmp/tidb_cdc_test/changefeed_auto_stop/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_auto_stop/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t48 not exists for 2-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d37103ef00008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz, pid:18121, start at 2024-05-06 19:57:25.829032103 +0800 CST m=+5.183666272 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:59:25.838 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:57:25.820 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:47:25.820 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d37103ef00008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz, pid:18121, start at 2024-05-06 19:57:25.829032103 +0800 CST m=+5.183666272 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:59:25.838 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:57:25.820 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:47:25.820 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d37103e180013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz, pid:18197, start at 2024-05-06 19:57:25.798786016 +0800 CST m=+5.085304034 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:59:25.804 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:57:25.766 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:47:25.766 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table move_table.usertable not exists for 2-th check, retry later + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.19502.out cli tso query --pd=http://127.0.0.1:2379 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/common_1/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:57:28 CST 2024] <<<<<< run test case common_1 success! >>>>>> table processor_delay.t48 not exists for 3-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:57:29 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/44895325-9073-4ccc-a781-2cb4067582f2 {"id":"44895325-9073-4ccc-a781-2cb4067582f2","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996645} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc3c3fa95 44895325-9073-4ccc-a781-2cb4067582f2 /tidb/cdc/default/default/upstream/7365854436976575284 {"id":7365854436976575284,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/44895325-9073-4ccc-a781-2cb4067582f2 {"id":"44895325-9073-4ccc-a781-2cb4067582f2","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996645} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc3c3fa95 44895325-9073-4ccc-a781-2cb4067582f2 /tidb/cdc/default/default/upstream/7365854436976575284 {"id":7365854436976575284,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/44895325-9073-4ccc-a781-2cb4067582f2 {"id":"44895325-9073-4ccc-a781-2cb4067582f2","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996645} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc3c3fa95 44895325-9073-4ccc-a781-2cb4067582f2 /tidb/cdc/default/default/upstream/7365854436976575284 {"id":7365854436976575284,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Mon May 6 19:57:29 CST 2024] <<<<<< START cdc server in changefeed_auto_stop case >>>>>> table move_table.usertable exists go: downloading github.com/go-mysql-org/go-mysql v1.7.1-0.20240314115043-2199dfb0ba98 go: downloading github.com/IBM/sarama v1.41.2 go: downloading gorm.io/gorm v1.24.5 go: downloading github.com/gavv/monotime v0.0.0-20190418164738-30dba4353424 go: downloading github.com/edwingeng/deque v0.0.0-20191220032131-8596380dee17 go: downloading github.com/xdg/scram v1.0.5 go: downloading github.com/benbjohnson/clock v1.3.5 go: downloading github.com/gin-gonic/gin v1.9.1 go: downloading github.com/tinylib/msgp v1.1.6 go: downloading go.etcd.io/etcd/server/v3 v3.5.12 go: downloading github.com/apache/pulsar-client-go v0.11.0 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/KimMachineGun/automemlimit v0.2.4 go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading github.com/cakturk/go-netstat v0.0.0-20200220111822-e5b49efee7a5 go: downloading github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2 go: downloading github.com/aws/aws-sdk-go-v2 v1.19.1 go: downloading github.com/stretchr/objx v0.5.2 go: downloading github.com/xdg/stringprep v1.0.3 go: downloading github.com/containerd/cgroups v1.0.4 go: downloading github.com/jinzhu/now v1.1.5 go: downloading github.com/jinzhu/inflection v1.0.0 go: downloading github.com/philhofer/fwd v1.1.1 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading github.com/cilium/ebpf v0.4.0 go: downloading github.com/godbus/dbus/v5 v5.0.4 go: downloading github.com/opencontainers/runtime-spec v1.0.2 go: downloading github.com/siddontang/go v0.0.0-20180604090527-bdc77568d726 go: downloading github.com/siddontang/go-log v0.0.0-20180807004314-8d05993dda07 + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/pipeline/ProcessorSyncResolvedError=1*return(true);github.com/pingcap/tiflow/cdc/processor/ProcessorUpdatePositionDelaying=sleep(1000)' + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_auto_stop.1178511787.out server --log-file /tmp/tidb_cdc_test/changefeed_auto_stop/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_auto_stop/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 go: downloading github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 go: downloading github.com/gin-contrib/sse v0.1.0 go: downloading github.com/ugorji/go/codec v1.2.11 go: downloading github.com/go-playground/validator/v10 v10.14.0 go: downloading github.com/pelletier/go-toml/v2 v2.0.8 go: downloading github.com/99designs/keyring v1.2.1 go: downloading golang.org/x/mod v0.17.0 go: downloading github.com/pierrec/lz4 v2.6.1+incompatible go: downloading github.com/bits-and-blooms/bitset v1.4.0 go: downloading github.com/linkedin/goavro/v2 v2.11.1 go: downloading github.com/AthenZ/athenz v1.10.39 go: downloading github.com/spaolacci/murmur3 v1.1.0 go: downloading github.com/aws/smithy-go v1.13.5 go: downloading github.com/dvsekhvalnov/jose2go v1.5.0 go: downloading github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c go: downloading github.com/mtibben/percent v0.2.1 go: downloading github.com/gabriel-vasile/mimetype v1.4.2 go: downloading github.com/go-playground/universal-translator v0.18.1 go: downloading github.com/leodido/go-urn v1.2.4 go: downloading github.com/go-playground/locales v0.14.1 go: downloading github.com/eapache/go-resiliency v1.4.0 go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 go: downloading github.com/jcmturner/gokrb5/v8 v8.4.4 go: downloading github.com/eapache/queue v1.1.0 go: downloading github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/jcmturner/gofork v1.7.6 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0 go: downloading github.com/soheilhy/cmux v0.1.5 go: downloading github.com/hashicorp/errwrap v1.0.0 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 go: downloading github.com/golang-jwt/jwt/v4 v4.5.0 go: downloading go.etcd.io/etcd/pkg/v3 v3.5.12 go: downloading go.etcd.io/etcd/raft/v3 v3.5.12 go: downloading go.etcd.io/bbolt v1.3.9 go: downloading github.com/jonboulle/clockwork v0.4.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.22.0 go: downloading github.com/gorilla/websocket v1.5.1 go: downloading go.etcd.io/etcd/client/v2 v2.305.12 go: downloading go.opentelemetry.io/otel/sdk v1.22.0 go: downloading github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 go: downloading go.opentelemetry.io/proto/otlp v1.1.0 go: downloading github.com/cenkalti/backoff/v4 v4.2.1 go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 go: downloading github.com/hashicorp/go-uuid v1.0.3 go: downloading github.com/jcmturner/rpc/v2 v2.0.3 go: downloading github.com/jcmturner/aescts/v2 v2.0.0 go: downloading github.com/jcmturner/dnsutils/v2 v2.0.0 + set +x + tso='449576081626234881 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576081626234881 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449576081626234881 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Mon May 6 19:57:30 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.1954319545.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 \033[0;36m<<< Run all test success >>>\033[0m table processor_delay.t48 not exists for 4-th check, retry later [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:57:32 GMT < Content-Length: 1299 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/44895325-9073-4ccc-a781-2cb4067582f2 {"id":"44895325-9073-4ccc-a781-2cb4067582f2","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996645} /tidb/cdc/default/__cdc_meta__/capture/e5bceb5f-a0bb-41f3-9fd5-31b1c2705971 {"id":"e5bceb5f-a0bb-41f3-9fd5-31b1c2705971","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996649} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc3c3fa95 44895325-9073-4ccc-a781-2cb4067582f2 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc3c3fad0 e5bceb5f-a0bb-41f3-9fd5-31b1c2705971 /tidb/cdc/default/default/upstream/7365854436976575284 {"id":7365854436976575284,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/44895325-9073-4ccc-a781-2cb4067582f2 {"id":"44895325-9073-4ccc-a781-2cb4067582f2","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996645} /tidb/cdc/default/__cdc_meta__/capture/e5bceb5f-a0bb-41f3-9fd5-31b1c2705971 {"id":"e5bceb5f-a0bb-41f3-9fd5-31b1c2705971","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996649} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc3c3fa95 44895325-9073-4ccc-a781-2cb4067582f2 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc3c3fad0 e5bceb5f-a0bb-41f3-9fd5-31b1c2705971 /tidb/cdc/default/default/upstream/7365854436976575284 {"id":7365854436976575284,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/44895325-9073-4ccc-a781-2cb4067582f2 {"id":"44895325-9073-4ccc-a781-2cb4067582f2","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996645} /tidb/cdc/default/__cdc_meta__/capture/e5bceb5f-a0bb-41f3-9fd5-31b1c2705971 {"id":"e5bceb5f-a0bb-41f3-9fd5-31b1c2705971","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996649} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc3c3fa95 44895325-9073-4ccc-a781-2cb4067582f2 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc3c3fad0 e5bceb5f-a0bb-41f3-9fd5-31b1c2705971 /tidb/cdc/default/default/upstream/7365854436976575284 {"id":7365854436976575284,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + workdir=/tmp/tidb_cdc_test/changefeed_auto_stop + sink_uri='file:///tmp/tidb_cdc_test/changefeed_auto_stop/storage_test/ticdc-changefeed-auto-stop-test-30538?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/changefeed_auto_stop ++ date + echo '[Mon May 6 19:57:32 CST 2024] <<<<<< START storage consumer in changefeed_auto_stop case >>>>>>' [Mon May 6 19:57:32 CST 2024] <<<<<< START storage consumer in changefeed_auto_stop case >>>>>> + cd /tmp/tidb_cdc_test/changefeed_auto_stop + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/changefeed_auto_stop + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/changefeed_auto_stop/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/changefeed_auto_stop/storage_test/ticdc-changefeed-auto-stop-test-30538?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' check_changefeed_state http://127.0.0.1:2379 a21cfb81-01da-45e5-8ccb-5bc056c7b05d normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=a21cfb81-01da-45e5-8ccb-5bc056c7b05d + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c a21cfb81-01da-45e5-8ccb-5bc056c7b05d -s + info='{ "upstream_id": 7365854436976575284, "namespace": "default", "id": "a21cfb81-01da-45e5-8ccb-5bc056c7b05d", "state": "normal", "checkpoint_tso": 449576079427108865, "checkpoint_time": "2024-05-06 19:57:20.881", "error": null }' + echo '{ "upstream_id": 7365854436976575284, "namespace": "default", "id": "a21cfb81-01da-45e5-8ccb-5bc056c7b05d", "state": "normal", "checkpoint_tso": 449576079427108865, "checkpoint_time": "2024-05-06 19:57:20.881", "error": null }' { "upstream_id": 7365854436976575284, "namespace": "default", "id": "a21cfb81-01da-45e5-8ccb-5bc056c7b05d", "state": "normal", "checkpoint_tso": 449576079427108865, "checkpoint_time": "2024-05-06 19:57:20.881", "error": null } ++ echo '{' '"upstream_id":' 7365854436976575284, '"namespace":' '"default",' '"id":' '"a21cfb81-01da-45e5-8ccb-5bc056c7b05d",' '"state":' '"normal",' '"checkpoint_tso":' 449576079427108865, '"checkpoint_time":' '"2024-05-06' '19:57:20.881",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7365854436976575284, '"namespace":' '"default",' '"id":' '"a21cfb81-01da-45e5-8ccb-5bc056c7b05d",' '"state":' '"normal",' '"checkpoint_tso":' 449576079427108865, '"checkpoint_time":' '"2024-05-06' '19:57:20.881",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully table changefeed_auto_stop_1.usertable not exists for 1-th check, retry later table processor_delay.t48 exists table processor_delay.t49 exists table processor_delay.t50 not exists for 1-th check, retry later go: downloading github.com/ardielle/ardielle-go v1.5.2 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:57:33 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9b8c2c6a-2660-4675-ab61-ef2faa4920a8 {"id":"9b8c2c6a-2660-4675-ab61-ef2faa4920a8","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996651} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc3e39439 9b8c2c6a-2660-4675-ab61-ef2faa4920a8 /tidb/cdc/default/default/upstream/7365854463711402869 {"id":7365854463711402869,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9b8c2c6a-2660-4675-ab61-ef2faa4920a8 {"id":"9b8c2c6a-2660-4675-ab61-ef2faa4920a8","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996651} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc3e39439 9b8c2c6a-2660-4675-ab61-ef2faa4920a8 /tidb/cdc/default/default/upstream/7365854463711402869 {"id":7365854463711402869,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9b8c2c6a-2660-4675-ab61-ef2faa4920a8 {"id":"9b8c2c6a-2660-4675-ab61-ef2faa4920a8","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996651} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc3e39439 9b8c2c6a-2660-4675-ab61-ef2faa4920a8 /tidb/cdc/default/default/upstream/7365854463711402869 {"id":7365854463711402869,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449576081626234881 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.19603.out cli changefeed create --start-ts=449576081626234881 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7365854463711402869,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-06T19:57:34.245949081+08:00","start_ts":449576081626234881,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576081626234881,"checkpoint_ts":449576081626234881,"checkpoint_time":"2024-05-06 19:57:29.270"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... table changefeed_auto_stop_1.usertable not exists for 2-th check, retry later table processor_delay.t50 not exists for 2-th check, retry later + set +x ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 3317 0 --:--:-- --:--:-- --:--:-- 3348 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-06 19:57:29.270","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-05-06 19:57:35.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '19:57:29.270","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-06' '19:57:35.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '19:57:29.270","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-06' '19:57:35.000","info":"Data' syncing is 'finished"}' ++ jq -r .sink_checkpoint_ts + sink_checkpoint_ts='2024-05-06 19:57:29.270' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '19:57:29.270","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-06' '19:57:35.000","info":"Data' syncing is 'finished"}' ++ jq -r .puller_resolved_ts + puller_resolved_ts='1970-01-01 08:00:00.000' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '19:57:29.270","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-06' '19:57:35.000","info":"Data' syncing is 'finished"}' ++ jq -r .last_synced_ts + last_synced_ts='1970-01-01 08:00:00.000' + '[' true '!=' true ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' ++ date '+%Y-%m-%d %H:%M:%S' + current='2024-05-06 19:57:35' + echo 'sink_checkpoint_ts is 2024-05-06' 19:57:29.270 sink_checkpoint_ts is 2024-05-06 19:57:29.270 ++ date -d '2024-05-06 19:57:29.270' +%s + checkpoint_timestamp=1714996649 ++ date -d '2024-05-06 19:57:35' +%s + current_timestamp=1714996655 + '[' 6 -gt 300 ']' + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table changefeed_auto_stop_1.usertable not exists for 3-th check, retry later table processor_delay.t50 not exists for 3-th check, retry later table test.t1 exists + sleep 5 table changefeed_auto_stop_1.usertable not exists for 4-th check, retry later table processor_delay.t50 not exists for 4-th check, retry later table changefeed_auto_stop_1.usertable not exists for 5-th check, retry later table processor_delay.t50 not exists for 5-th check, retry later table changefeed_auto_stop_1.usertable exists table changefeed_auto_stop_2.usertable not exists for 1-th check, retry later ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 1685 0 --:--:-- --:--:-- --:--:-- 1687 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-06 19:57:42.721","puller_resolved_ts":"2024-05-06 19:57:35.920","last_synced_ts":"2024-05-06 19:57:35.971","now_ts":"2024-05-06 19:57:43.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '19:57:42.721","puller_resolved_ts":"2024-05-06' '19:57:35.920","last_synced_ts":"2024-05-06' '19:57:35.971","now_ts":"2024-05-06' '19:57:43.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '19:57:42.721","puller_resolved_ts":"2024-05-06' '19:57:35.920","last_synced_ts":"2024-05-06' '19:57:35.971","now_ts":"2024-05-06' '19:57:43.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 table processor_delay.t50 exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... table changefeed_auto_stop_2.usertable exists table changefeed_auto_stop_3.usertable not exists for 1-th check, retry later wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:57:45 CST 2024] <<<<<< run test case processor_etcd_worker_delay success! >>>>>> table changefeed_auto_stop_3.usertable not exists for 2-th check, retry later table changefeed_auto_stop_3.usertable not exists for 3-th check, retry later table changefeed_auto_stop_3.usertable not exists for 4-th check, retry later table changefeed_auto_stop_3.usertable not exists for 5-th check, retry later table changefeed_auto_stop_3.usertable exists table changefeed_auto_stop_4.usertable exists check diff failed 1-th time, retry later check diff failed 2-th time, retry later + '[' storage == mysql ']' + stop_tidb_cluster check diff failed 3-th time, retry later [2024/05/06 19:57:57.890 +08:00] [INFO] [main.go:61] ["table mover started"] [2024/05/06 19:57:57.897 +08:00] [INFO] [main.go:166] ["new cluster initialized"] [2024/05/06 19:57:57.898 +08:00] [DEBUG] [main.go:192] ["retrieved owner ID"] [ownerID=b92fe9df-19db-4d86-9166-94f2b1f56ac8] [2024/05/06 19:57:57.898 +08:00] [DEBUG] [main.go:199] ["retrieved owner addr"] [ownerAddr=127.0.0.1:8300] [2024/05/06 19:57:57.899 +08:00] [DEBUG] [main.go:210] ["retrieved changefeeds"] [changefeedsError="json: unsupported type: map[model.ChangeFeedID]*mvccpb.KeyValue"] [2024/05/06 19:57:58.095 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d] [captureID=03134e50-ddf5-4cfb-9d25-792d52f5d3dd] [processorDetail="{\"table_ids\":[]}"] [2024/05/06 19:57:58.294 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d] [captureID=761a6499-cd6d-4a2c-afd1-7fea1e2d348f] [processorDetail="{\"table_ids\":[]}"] [2024/05/06 19:57:58.496 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d] [captureID=b92fe9df-19db-4d86-9166-94f2b1f56ac8] [processorDetail="{\"table_ids\":[106,108]}"] [2024/05/06 19:57:58.496 +08:00] [INFO] [main.go:75] ["task status"] [status="{\"03134e50-ddf5-4cfb-9d25-792d52f5d3dd\":[],\"761a6499-cd6d-4a2c-afd1-7fea1e2d348f\":[],\"b92fe9df-19db-4d86-9166-94f2b1f56ac8\":[{\"ID\":106,\"Changefeed\":\"6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d\"},{\"ID\":108,\"Changefeed\":\"6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d\"}]}"] [2024/05/06 19:57:58.497 +08:00] [DEBUG] [main.go:288] ["preparing HTTP API call to owner"] [formStr="cf-id=6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d&target-cp-id=761a6499-cd6d-4a2c-afd1-7fea1e2d348f&table-id=106"] [2024/05/06 19:57:58.544 +08:00] [INFO] [main.go:180] ["moved table successful"] [tableID=106] [2024/05/06 19:57:58.544 +08:00] [DEBUG] [main.go:288] ["preparing HTTP API call to owner"] [formStr="cf-id=6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d&target-cp-id=761a6499-cd6d-4a2c-afd1-7fea1e2d348f&table-id=108"] [2024/05/06 19:57:58.598 +08:00] [INFO] [main.go:180] ["moved table successful"] [tableID=108] [2024/05/06 19:57:58.598 +08:00] [INFO] [main.go:114] ["all tables are moved"] [sourceCapture=b92fe9df-19db-4d86-9166-94f2b1f56ac8] [targetCapture=761a6499-cd6d-4a2c-afd1-7fea1e2d348f] table move_table.check1 exists check diff successfully check diff failed 4-th time, retry later check diff successfully =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/sink_hang/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... wait process cdc.test exit for 1-th time... + stop_tidb_cluster wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... [2024/05/06 19:58:05.696 +08:00] [INFO] [main.go:61] ["table mover started"] [2024/05/06 19:58:05.700 +08:00] [INFO] [main.go:166] ["new cluster initialized"] [2024/05/06 19:58:05.701 +08:00] [DEBUG] [main.go:192] ["retrieved owner ID"] [ownerID=b92fe9df-19db-4d86-9166-94f2b1f56ac8] [2024/05/06 19:58:05.705 +08:00] [DEBUG] [main.go:199] ["retrieved owner addr"] [ownerAddr=127.0.0.1:8300] [2024/05/06 19:58:05.708 +08:00] [DEBUG] [main.go:210] ["retrieved changefeeds"] [changefeedsError="json: unsupported type: map[model.ChangeFeedID]*mvccpb.KeyValue"] [2024/05/06 19:58:05.894 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d] [captureID=03134e50-ddf5-4cfb-9d25-792d52f5d3dd] [processorDetail="{\"table_ids\":[]}"] cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:58:05 CST 2024] <<<<<< run test case changefeed_auto_stop success! >>>>>> [2024/05/06 19:58:06.093 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d] [captureID=761a6499-cd6d-4a2c-afd1-7fea1e2d348f] [processorDetail="{\"table_ids\":[108]}"] [2024/05/06 19:58:06.292 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d] [captureID=b92fe9df-19db-4d86-9166-94f2b1f56ac8] [processorDetail="{\"table_ids\":[110]}"] [2024/05/06 19:58:06.292 +08:00] [INFO] [main.go:75] ["task status"] [status="{\"03134e50-ddf5-4cfb-9d25-792d52f5d3dd\":[],\"761a6499-cd6d-4a2c-afd1-7fea1e2d348f\":[{\"ID\":108,\"Changefeed\":\"6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d\"}],\"b92fe9df-19db-4d86-9166-94f2b1f56ac8\":[{\"ID\":110,\"Changefeed\":\"6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d\"}]}"] [2024/05/06 19:58:06.292 +08:00] [DEBUG] [main.go:288] ["preparing HTTP API call to owner"] [formStr="cf-id=6b903251-7b4e-4c22-b6ee-c25bdf7c3b5d&target-cp-id=b92fe9df-19db-4d86-9166-94f2b1f56ac8&table-id=108"] [2024/05/06 19:58:06.343 +08:00] [INFO] [main.go:180] ["moved table successful"] [tableID=108] [2024/05/06 19:58:06.343 +08:00] [INFO] [main.go:114] ["all tables are moved"] [sourceCapture=761a6499-cd6d-4a2c-afd1-7fea1e2d348f] [targetCapture=b92fe9df-19db-4d86-9166-94f2b1f56ac8] start tidb cluster in /tmp/tidb_cdc_test/sink_hang Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } check diff failed 1-th time, retry later [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff successfully table move_table.check2 not exists for 1-th check, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table move_table.check2 not exists for 2-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table move_table.check2 not exists for 3-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table move_table.check2 not exists for 4-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d371347bc0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-81vjm, pid:8314, start at 2024-05-06 19:58:15.567829595 +0800 CST m=+5.143425009 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:00:15.576 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:58:15.535 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:48:15.535 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d371347bc0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-81vjm, pid:8314, start at 2024-05-06 19:58:15.567829595 +0800 CST m=+5.143425009 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:00:15.576 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:58:15.535 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:48:15.535 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d371349fc0015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-81vjm, pid:8400, start at 2024-05-06 19:58:15.71911048 +0800 CST m=+5.245478953 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:00:15.727 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:58:15.730 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:48:15.730 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sink_hang/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sink_hang/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } table move_table.check2 not exists for 5-th check, retry later [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Mon May 6 19:58:18 CST 2024] <<<<<< START cdc server in sink_hang case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkExecDMLError=2*return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_hang.96499651.out server --log-file /tmp/tidb_cdc_test/sink_hang/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sink_hang/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table move_table.check2 exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:58:21 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c63a7a32-a0a3-484a-a235-81a4e2c540ac {"id":"c63a7a32-a0a3-484a-a235-81a4e2c540ac","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996699} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc4adf92c c63a7a32-a0a3-484a-a235-81a4e2c540ac /tidb/cdc/default/default/upstream/7365854686097344348 {"id":7365854686097344348,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c63a7a32-a0a3-484a-a235-81a4e2c540ac {"id":"c63a7a32-a0a3-484a-a235-81a4e2c540ac","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996699} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc4adf92c c63a7a32-a0a3-484a-a235-81a4e2c540ac /tidb/cdc/default/default/upstream/7365854686097344348 {"id":7365854686097344348,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c63a7a32-a0a3-484a-a235-81a4e2c540ac {"id":"c63a7a32-a0a3-484a-a235-81a4e2c540ac","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996699} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc4adf92c c63a7a32-a0a3-484a-a235-81a4e2c540ac /tidb/cdc/default/default/upstream/7365854686097344348 {"id":7365854686097344348,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + workdir=/tmp/tidb_cdc_test/sink_hang + sink_uri='file:///tmp/tidb_cdc_test/sink_hang/storage_test/ticdc-sink-hang-test-16597?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/sink_hang ++ date + echo '[Mon May 6 19:58:22 CST 2024] <<<<<< START storage consumer in sink_hang case >>>>>>' [Mon May 6 19:58:22 CST 2024] <<<<<< START storage consumer in sink_hang case >>>>>> + cd /tmp/tidb_cdc_test/sink_hang + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/sink_hang + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/sink_hang/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/sink_hang/storage_test/ticdc-sink-hang-test-16597?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' wait process cdc.test exit for 3-th time... check_changefeed_status 127.0.0.1:8300 37d37dc7-e038-4356-82bf-1e9096eed611 normal last_error null + endpoint=127.0.0.1:8300 + changefeed_id=37d37dc7-e038-4356-82bf-1e9096eed611 + expected_state=normal + field=last_error + error_pattern=null ++ curl 127.0.0.1:8300/api/v2/changefeeds/37d37dc7-e038-4356-82bf-1e9096eed611/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 86 100 86 0 0 744 0 --:--:-- --:--:-- --:--:-- 747 + info='{"state":"normal","resolved_ts":449576095418941446,"checkpoint_ts":449576095418941446}' + echo '{"state":"normal","resolved_ts":449576095418941446,"checkpoint_ts":449576095418941446}' {"state":"normal","resolved_ts":449576095418941446,"checkpoint_ts":449576095418941446} ++ echo '{"state":"normal","resolved_ts":449576095418941446,"checkpoint_ts":449576095418941446}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] + [[ -z last_error ]] ++ echo '{"state":"normal","resolved_ts":449576095418941446,"checkpoint_ts":449576095418941446}' ++ jq -r .last_error.message + error_msg=null + [[ ! null =~ null ]] run task successfully check_changefeed_status 127.0.0.1:8300 37d37dc7-e038-4356-82bf-1e9096eed611 normal last_warning null + endpoint=127.0.0.1:8300 + changefeed_id=37d37dc7-e038-4356-82bf-1e9096eed611 + expected_state=normal + field=last_warning + error_pattern=null ++ curl 127.0.0.1:8300/api/v2/changefeeds/37d37dc7-e038-4356-82bf-1e9096eed611/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 86 100 86 0 0 656 0 --:--:-- --:--:-- --:--:-- 661 + info='{"state":"normal","resolved_ts":449576095418941446,"checkpoint_ts":449576095418941446}' + echo '{"state":"normal","resolved_ts":449576095418941446,"checkpoint_ts":449576095418941446}' {"state":"normal","resolved_ts":449576095418941446,"checkpoint_ts":449576095418941446} ++ echo '{"state":"normal","resolved_ts":449576095418941446,"checkpoint_ts":449576095418941446}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] + [[ -z last_warning ]] ++ echo '{"state":"normal","resolved_ts":449576095418941446,"checkpoint_ts":449576095418941446}' ++ jq -r .last_warning.message + error_msg=null + [[ ! null =~ null ]] run task successfully table sink_hang.t1 not exists for 1-th check, retry later wait process cdc.test exit for 4-th time... cdc.test: no process found wait process cdc.test exit for 5-th time... process cdc.test already exit [Mon May 6 19:58:23 CST 2024] <<<<<< run test case move_table success! >>>>>> table sink_hang.t1 not exists for 2-th check, retry later table sink_hang.t1 not exists for 3-th check, retry later table sink_hang.t1 not exists for 4-th check, retry later table sink_hang.t1 not exists for 5-th check, retry later table sink_hang.t1 exists table sink_hang.t2 exists check diff failed 1-th time, retry later check diff failed 2-th time, retry later check diff failed 3-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/autorandom/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... check diff failed 4-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/autorandom Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... check diff failed 5-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Mon May 6 19:58:44 CST 2024] <<<<<< run test case sink_hang success! >>>>>> Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3715c5e80009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-pst3l, pid:21234, start at 2024-05-06 19:58:56.410137643 +0800 CST m=+10.815397588 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:00:56.438 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:58:56.478 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:48:56.478 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3715c5e80009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-pst3l, pid:21234, start at 2024-05-06 19:58:56.410137643 +0800 CST m=+10.815397588 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:00:56.438 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:58:56.478 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:48:56.478 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3715c6500007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-pst3l, pid:21261, start at 2024-05-06 19:58:56.440477873 +0800 CST m=+10.748661239 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:00:56.490 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:58:56.508 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:48:56.508 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/autorandom/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/autorandom/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Mon May 6 19:59:00 CST 2024] <<<<<< START cdc server in autorandom case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.autorandom.2248422486.out server --log-file /tmp/tidb_cdc_test/autorandom/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/autorandom/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:59:03 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2388ea36-ba76-4a67-b66b-430bab2ad0a2 {"id":"2388ea36-ba76-4a67-b66b-430bab2ad0a2","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996740} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc536fa3c 2388ea36-ba76-4a67-b66b-430bab2ad0a2 /tidb/cdc/default/default/upstream/7365854832335680248 {"id":7365854832335680248,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2388ea36-ba76-4a67-b66b-430bab2ad0a2 {"id":"2388ea36-ba76-4a67-b66b-430bab2ad0a2","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996740} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc536fa3c 2388ea36-ba76-4a67-b66b-430bab2ad0a2 /tidb/cdc/default/default/upstream/7365854832335680248 {"id":7365854832335680248,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2388ea36-ba76-4a67-b66b-430bab2ad0a2 {"id":"2388ea36-ba76-4a67-b66b-430bab2ad0a2","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996740} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc536fa3c 2388ea36-ba76-4a67-b66b-430bab2ad0a2 /tidb/cdc/default/default/upstream/7365854832335680248 {"id":7365854832335680248,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: cfcaef2c-3ee6-44ff-a9e3-1e818461b334 Info: {"upstream_id":7365854832335680248,"namespace":"default","id":"cfcaef2c-3ee6-44ff-a9e3-1e818461b334","sink_uri":"file:///tmp/tidb_cdc_test/autorandom/storage_test/ticdc-autorandom-test-7233?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:59:03.682625267+08:00","start_ts":449576106322296837,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576106322296837,"checkpoint_ts":449576106322296837,"checkpoint_time":"2024-05-06 19:59:03.478"} + workdir=/tmp/tidb_cdc_test/autorandom + sink_uri='file:///tmp/tidb_cdc_test/autorandom/storage_test/ticdc-autorandom-test-7233?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/autorandom ++ date + echo '[Mon May 6 19:59:03 CST 2024] <<<<<< START storage consumer in autorandom case >>>>>>' [Mon May 6 19:59:03 CST 2024] <<<<<< START storage consumer in autorandom case >>>>>> + cd /tmp/tidb_cdc_test/autorandom + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/autorandom + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/autorandom/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/autorandom/storage_test/ticdc-autorandom-test-7233?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table autorandom_test.table_a not exists for 1-th check, retry later table autorandom_test.table_a not exists for 2-th check, retry later table autorandom_test.table_a not exists for 3-th check, retry later table autorandom_test.table_a not exists for 4-th check, retry later table autorandom_test.table_a not exists for 5-th check, retry later table autorandom_test.table_a exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:59:17 CST 2024] <<<<<< run test case autorandom success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/generate_column/run.sh using Sink-Type: storage... <<================= [Mon May 6 19:59:30 CST 2024] <<<<<< run test case generate_column success! >>>>>> \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 8731 0 --:--:-- --:--:-- --:--:-- 8817 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-06 19:57:21.352","puller_resolved_ts":"2024-05-06 19:57:21.352","last_synced_ts":"2024-05-06 19:57:13.951","now_ts":"2024-05-06 19:59:32.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '19:57:21.352","puller_resolved_ts":"2024-05-06' '19:57:21.352","last_synced_ts":"2024-05-06' '19:57:13.951","now_ts":"2024-05-06' '19:59:32.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ jq -r .info ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '19:57:21.352","puller_resolved_ts":"2024-05-06' '19:57:21.352","last_synced_ts":"2024-05-06' '19:57:13.951","now_ts":"2024-05-06' '19:59:32.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tidb conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status_with_redo + mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3719bcf0001b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l, pid:15534, start at 2024-05-06 20:00:01.385710281 +0800 CST m=+5.272207091 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:02:01.395 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-20:00:01.390 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:50:01.390 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3719bcf0001b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l, pid:15534, start at 2024-05-06 20:00:01.385710281 +0800 CST m=+5.272207091 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:02:01.395 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-20:00:01.390 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:50:01.390 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3719be680014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l, pid:15606, start at 2024-05-06 20:00:01.471576247 +0800 CST m=+5.311804881 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:02:01.478 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-20:00:01.434 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:50:01.434 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2577 0 --:--:-- --:--:-- --:--:-- 2600 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-06 19:59:52.870","puller_resolved_ts":"2024-05-06 19:59:44.821","last_synced_ts":"2024-05-06 19:57:35.971","now_ts":"2024-05-06 19:59:53.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '19:59:52.870","puller_resolved_ts":"2024-05-06' '19:59:44.821","last_synced_ts":"2024-05-06' '19:57:35.971","now_ts":"2024-05-06' '19:59:53.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' + kill_pd ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 17408 7.1 0.0 13848796 141524 ? Sl 19:57 0:11 pd-server --advertise-client-urls http://127.0.0.1:2379 --client-urls http://0.0.0.0:2379 --advertise-peer-urls http://127.0.0.1:2380 --peer-urls http://0.0.0.0:2380 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/pd1.log --data-dir /tmp/tidb_cdc_test/synced_status/pd1 --name=pd1 --initial-cluster=pd1=http://127.0.0.1:2380 jenkins 17478 4.9 0.0 13390812 141660 ? Sl 19:57 0:07 pd-server --advertise-client-urls http://127.0.0.1:2479 --client-urls http://0.0.0.0:2479 --advertise-peer-urls http://127.0.0.1:2480 --peer-urls http://0.0.0.0:2480 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/down_pd.log --data-dir /tmp/tidb_cdc_test/synced_status/down_pd' ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 + sleep 20 {"level":"warn","ts":1714996800.8337374,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002b3ac40/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1714996800.8337917,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1714996800.8665802,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0010dc380/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714996800.8666227,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1714996800.89436,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002310fc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714996800.894412,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-05-06T20:00:05.719539+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ff8000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T20:00:05.720816+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001402000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T20:00:05.807692+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0006bcfc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} + cd /tmp/tidb_cdc_test/synced_status_with_redo ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.16847.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449576122906312706 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576122906312706 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449576122906312706 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test [Mon May 6 20:00:08 CST 2024] <<<<<< START cdc server in synced_status_with_redo case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.1688216884.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 12:00:11 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/78e85a11-bcaf-4e73-a947-57699b632182 {"id":"78e85a11-bcaf-4e73-a947-57699b632182","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996808} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc641823b 78e85a11-bcaf-4e73-a947-57699b632182 /tidb/cdc/default/default/upstream/7365855130967455457 {"id":7365855130967455457,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/78e85a11-bcaf-4e73-a947-57699b632182 {"id":"78e85a11-bcaf-4e73-a947-57699b632182","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996808} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc641823b 78e85a11-bcaf-4e73-a947-57699b632182 /tidb/cdc/default/default/upstream/7365855130967455457 {"id":7365855130967455457,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/78e85a11-bcaf-4e73-a947-57699b632182 {"id":"78e85a11-bcaf-4e73-a947-57699b632182","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996808} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc641823b 78e85a11-bcaf-4e73-a947-57699b632182 /tidb/cdc/default/default/upstream/7365855130967455457 {"id":7365855130967455457,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449576122906312706 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.16942.out cli changefeed create --start-ts=449576122906312706 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7365855130967455457,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-06T20:00:11.83309791+08:00","start_ts":449576122906312706,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576122906312706,"checkpoint_ts":449576122906312706,"checkpoint_time":"2024-05-06 20:00:06.741"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... {"level":"warn","ts":"2024-05-06T20:00:11.720676+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ff8000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T20:00:11.72273+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001402000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T20:00:11.808876+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0006bcfc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0{"level":"warn","ts":"2024-05-06T20:00:17.721789+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ff8000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T20:00:17.723746+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001402000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T20:00:17.810796+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0006bcfc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} + kill_tidb ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo + info='jenkins 15530 3.5 0.0 2635508 195496 ? Sl 19:59 0:00 tidb-server -P 4000 -config /tmp/tidb_cdc_test/synced_status_with_redo/tidb-config-1714996796105031799.toml --store tikv --path 127.0.0.1:2379 --status=10080 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tidb.log jenkins 15534 11.6 0.0 2728708 260920 ? Sl 19:59 0:02 tidb-server -P 4001 -config /tmp/tidb_cdc_test/synced_status_with_redo/tidb-config-1714996796107990121.toml --store tikv --path 127.0.0.1:2379 --status=10081 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tidb_other.log jenkins 15606 12.3 0.0 2572792 257500 ? Sl 19:59 0:03 tidb-server -P 3306 -config /tmp/tidb_cdc_test/synced_status_with_redo/tidb-config-1714996796153531760.toml --store tikv --path 127.0.0.1:2479 --status=20080 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tidb_down.log' ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 3549 0 --:--:-- --:--:-- --:--:-- 3573 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-06 20:00:19.040","puller_resolved_ts":"2024-05-06 20:00:13.341","last_synced_ts":"2024-05-06 20:00:13.840","now_ts":"2024-05-06 20:00:20.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '20:00:19.040","puller_resolved_ts":"2024-05-06' '20:00:13.341","last_synced_ts":"2024-05-06' '20:00:13.840","now_ts":"2024-05-06' '20:00:20.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '20:00:19.040","puller_resolved_ts":"2024-05-06' '20:00:13.341","last_synced_ts":"2024-05-06' '20:00:13.840","now_ts":"2024-05-06' '20:00:20.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0{"level":"warn","ts":"2024-05-06T20:00:23.722917+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ff8000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T20:00:23.725141+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001402000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T20:00:23.812204+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0006bcfc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0{"level":"warn","ts":"2024-05-06T20:00:25.712558+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001402000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2024-05-06T20:00:25.712594+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-05-06T20:00:25.712619+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ff8000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2024-05-06T20:00:25.712652+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-05-06T20:00:25.765115+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0006bcfc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":"2024-05-06T20:00:25.765163+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:14 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:15 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0{"level":"warn","ts":"2024-05-06T20:00:29.723621+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ff8000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T20:00:29.726235+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001402000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T20:00:29.813373+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0006bcfc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:17 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:18 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:21 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:22 --:--:-- 0{"level":"warn","ts":"2024-05-06T20:00:35.724845+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ff8000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T20:00:35.726862+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001402000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T20:00:35.814598+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0006bcfc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"warn","ts":1714996835.8350904,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002b3ac40/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1714996835.8351243,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1714996835.8675942,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0010dc380/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714996835.8676286,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1714996835.8952653,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002310fc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714996835.8952942,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:23 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:24 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:25 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:26 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:27 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:28 --:--:-- 0{"level":"warn","ts":"2024-05-06T20:00:41.72582+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ff8000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T20:00:41.727953+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001402000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T20:00:41.81596+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0006bcfc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 27 + synced_status='{ "error_msg": "[CDC:ErrPDEtcdAPIError]etcd api call error: context deadline exceeded", "error_code": "CDC:ErrPDEtcdAPIError" }' ++ echo '{' '"error_msg":' '"[CDC:ErrPDEtcdAPIError]etcd' api call error: context deadline 'exceeded",' '"error_code":' '"CDC:ErrPDEtcdAPIError"' '}' ++ jq -r .error_code + error_code=CDC:ErrPDEtcdAPIError + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tikv conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d371de7cc0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz, pid:20712, start at 2024-05-06 20:01:09.643046882 +0800 CST m=+5.135043570 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:03:09.651 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-20:01:09.619 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:51:09.619 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d371de7cc0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz, pid:20712, start at 2024-05-06 20:01:09.643046882 +0800 CST m=+5.135043570 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:03:09.651 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-20:01:09.619 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:51:09.619 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d371de9f80012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz, pid:20793, start at 2024-05-06 20:01:09.775772334 +0800 CST m=+5.215146498 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:03:09.786 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-20:01:09.758 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:51:09.758 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.22150.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449576140372180994 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576140372180994 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449576140372180994 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Mon May 6 20:01:14 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2218822190.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 12:01:17 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/338fae14-15b5-4d70-83d4-2cccd5d3b49f {"id":"338fae14-15b5-4d70-83d4-2cccd5d3b49f","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996875} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc7521533 338fae14-15b5-4d70-83d4-2cccd5d3b49f /tidb/cdc/default/default/upstream/7365855433662661733 {"id":7365855433662661733,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/338fae14-15b5-4d70-83d4-2cccd5d3b49f {"id":"338fae14-15b5-4d70-83d4-2cccd5d3b49f","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996875} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc7521533 338fae14-15b5-4d70-83d4-2cccd5d3b49f /tidb/cdc/default/default/upstream/7365855433662661733 {"id":7365855433662661733,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/338fae14-15b5-4d70-83d4-2cccd5d3b49f {"id":"338fae14-15b5-4d70-83d4-2cccd5d3b49f","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996875} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc7521533 338fae14-15b5-4d70-83d4-2cccd5d3b49f /tidb/cdc/default/default/upstream/7365855433662661733 {"id":7365855433662661733,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449576140372180994 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.22250.out cli changefeed create --start-ts=449576140372180994 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7365855433662661733,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-06T20:01:18.384362258+08:00","start_ts":449576140372180994,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576140372180994,"checkpoint_ts":449576140372180994,"checkpoint_time":"2024-05-06 20:01:13.368"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 + kill_tikv ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 20193 27.1 0.4 3994596 1927820 ? Sl 20:01 0:06 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20160 --status-addr 127.0.0.1:20181 --log-file /tmp/tidb_cdc_test/synced_status/tikv1.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv1 jenkins 20194 20.5 0.4 3972576 1884268 ? Sl 20:01 0:04 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20161 --status-addr 127.0.0.1:20182 --log-file /tmp/tidb_cdc_test/synced_status/tikv2.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv2 jenkins 20195 20.5 0.4 3961316 1859308 ? Sl 20:01 0:04 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20162 --status-addr 127.0.0.1:20183 --log-file /tmp/tidb_cdc_test/synced_status/tikv3.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv3 jenkins 20197 26.7 0.4 3997668 1919016 ? Sl 20:01 0:06 tikv-server --pd 127.0.0.1:2479 -A 127.0.0.1:21160 --status-addr 127.0.0.1:21180 --log-file /tmp/tidb_cdc_test/synced_status/tikv_down.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv_down' ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 3347 0 --:--:-- --:--:-- --:--:-- 3375 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-06 20:01:26.568","puller_resolved_ts":"2024-05-06 20:01:19.819","last_synced_ts":"2024-05-06 20:01:19.918","now_ts":"2024-05-06 20:01:27.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '20:01:26.568","puller_resolved_ts":"2024-05-06' '20:01:19.819","last_synced_ts":"2024-05-06' '20:01:19.918","now_ts":"2024-05-06' '20:01:27.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '20:01:26.568","puller_resolved_ts":"2024-05-06' '20:01:19.819","last_synced_ts":"2024-05-06' '20:01:19.918","now_ts":"2024-05-06' '20:01:27.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2698 0 --:--:-- --:--:-- --:--:-- 2728 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-06 20:02:28.990","puller_resolved_ts":"2024-05-06 20:02:22.990","last_synced_ts":"2024-05-06 20:00:13.840","now_ts":"2024-05-06 20:02:31.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '20:02:28.990","puller_resolved_ts":"2024-05-06' '20:02:22.990","last_synced_ts":"2024-05-06' '20:00:13.840","now_ts":"2024-05-06' '20:02:31.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '20:02:28.990","puller_resolved_ts":"2024-05-06' '20:02:22.990","last_synced_ts":"2024-05-06' '20:00:13.840","now_ts":"2024-05-06' '20:02:31.000","info":"Data' syncing is 'finished"}' ++ jq -r .info + info='Data syncing is finished' + target_message='Data syncing is finished' + '[' 'Data syncing is finished' '!=' 'Data syncing is finished' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_failpoint conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status_with_redo + mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3724a66c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l, pid:18103, start at 2024-05-06 20:03:00.159573406 +0800 CST m=+5.303715736 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:05:00.166 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-20:03:00.173 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:53:00.173 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3724a66c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l, pid:18103, start at 2024-05-06 20:03:00.159573406 +0800 CST m=+5.303715736 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:05:00.166 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-20:03:00.173 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:53:00.173 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3724a72c0003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-rk36l, pid:18182, start at 2024-05-06 20:03:00.178070955 +0800 CST m=+5.257229533 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:05:00.185 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-20:03:00.171 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:53:00.171 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status_with_redo + export 'GO_FAILPOINTS=github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.19422.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449576169772679169 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576169772679169 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449576169772679169 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test [Mon May 6 20:03:06 CST 2024] <<<<<< START cdc server in synced_status_with_redo case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.1946519467.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 12:03:10 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7c6652b7-7ca2-4d9e-9192-3d4424ba7e85 {"id":"7c6652b7-7ca2-4d9e-9192-3d4424ba7e85","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996987} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc8f9bb37 7c6652b7-7ca2-4d9e-9192-3d4424ba7e85 /tidb/cdc/default/default/upstream/7365855900790092983 {"id":7365855900790092983,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7c6652b7-7ca2-4d9e-9192-3d4424ba7e85 {"id":"7c6652b7-7ca2-4d9e-9192-3d4424ba7e85","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996987} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc8f9bb37 7c6652b7-7ca2-4d9e-9192-3d4424ba7e85 /tidb/cdc/default/default/upstream/7365855900790092983 {"id":7365855900790092983,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7c6652b7-7ca2-4d9e-9192-3d4424ba7e85 {"id":"7c6652b7-7ca2-4d9e-9192-3d4424ba7e85","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714996987} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc8f9bb37 7c6652b7-7ca2-4d9e-9192-3d4424ba7e85 /tidb/cdc/default/default/upstream/7365855900790092983 {"id":7365855900790092983,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449576169772679169 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.19516.out cli changefeed create --start-ts=449576169772679169 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7365855900790092983,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-06T20:03:10.510975361+08:00","start_ts":449576169772679169,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576169772679169,"checkpoint_ts":449576169772679169,"checkpoint_time":"2024-05-06 20:03:05.522"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... + set +x + sleep 20 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 6498 0 --:--:-- --:--:-- --:--:-- 6513 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-06 20:03:05.522","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-05-06 20:03:32.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '20:03:05.522","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-06' '20:03:32.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ jq -r .info ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '20:03:05.522","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-06' '20:03:32.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + export GO_FAILPOINTS= + GO_FAILPOINTS= + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit + stop_tidb_cluster + check_logs /tmp/tidb_cdc_test/synced_status_with_redo ++ date + echo '[Mon May 6 20:03:43 CST 2024] <<<<<< run test case synced_status_with_redo success! >>>>>>' [Mon May 6 20:03:43 CST 2024] <<<<<< run test case synced_status_with_redo success! >>>>>> + stop_tidb_cluster ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 9149 0 --:--:-- --:--:-- --:--:-- 9269 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-06 20:01:26.568","puller_resolved_ts":"2024-05-06 20:01:26.568","last_synced_ts":"2024-05-06 20:01:19.918","now_ts":"2024-05-06 20:03:37.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '20:01:26.568","puller_resolved_ts":"2024-05-06' '20:01:26.568","last_synced_ts":"2024-05-06' '20:01:19.918","now_ts":"2024-05-06' '20:03:37.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '20:01:26.568","puller_resolved_ts":"2024-05-06' '20:01:26.568","last_synced_ts":"2024-05-06' '20:01:19.918","now_ts":"2024-05-06' '20:03:37.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tidb conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d372875f0000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz, pid:23429, start at 2024-05-06 20:04:02.569301675 +0800 CST m=+5.087500934 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:06:02.577 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-20:04:02.556 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:54:02.556 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d372875f0000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz, pid:23429, start at 2024-05-06 20:04:02.569301675 +0800 CST m=+5.087500934 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:06:02.577 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-20:04:02.556 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:54:02.556 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d372878340015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz, pid:23517, start at 2024-05-06 20:04:02.726383328 +0800 CST m=+5.192673093 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:06:02.734 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-20:04:02.701 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:54:02.701 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.24782.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449576185654411265 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576185654411265 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449576185654411265 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Mon May 6 20:04:07 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2482424826.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 12:04:10 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d5636a56-5906-46b4-99b9-7b5d1db0ae46 {"id":"d5636a56-5906-46b4-99b9-7b5d1db0ae46","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714997047} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc9f1b739 d5636a56-5906-46b4-99b9-7b5d1db0ae46 /tidb/cdc/default/default/upstream/7365856167313849580 {"id":7365856167313849580,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d5636a56-5906-46b4-99b9-7b5d1db0ae46 {"id":"d5636a56-5906-46b4-99b9-7b5d1db0ae46","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714997047} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc9f1b739 d5636a56-5906-46b4-99b9-7b5d1db0ae46 /tidb/cdc/default/default/upstream/7365856167313849580 {"id":7365856167313849580,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d5636a56-5906-46b4-99b9-7b5d1db0ae46 {"id":"d5636a56-5906-46b4-99b9-7b5d1db0ae46","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714997047} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc9f1b739 d5636a56-5906-46b4-99b9-7b5d1db0ae46 /tidb/cdc/default/default/upstream/7365856167313849580 {"id":7365856167313849580,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449576185654411265 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.24884.out cli changefeed create --start-ts=449576185654411265 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7365856167313849580,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-06T20:04:11.067183643+08:00","start_ts":449576185654411265,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576185654411265,"checkpoint_ts":449576185654411265,"checkpoint_time":"2024-05-06 20:04:06.106"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 + kill_tidb ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 23429 11.4 0.0 2744652 250120 ? Sl 20:03 0:02 tidb-server -P 4000 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1714997037475061160.toml --store tikv --path 127.0.0.1:2379 --status=10080 --log-file /tmp/tidb_cdc_test/synced_status/tidb.log jenkins 23433 3.4 0.0 2585756 186816 ? Sl 20:03 0:00 tidb-server -P 4001 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1714997037478496969.toml --store tikv --path 127.0.0.1:2379 --status=10081 --log-file /tmp/tidb_cdc_test/synced_status/tidb_other.log jenkins 23517 12.9 0.0 2869340 267220 ? Sl 20:03 0:02 tidb-server -P 3306 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1714997037526916408.toml --store tikv --path 127.0.0.1:2479 --status=20080 --log-file /tmp/tidb_cdc_test/synced_status/tidb_down.log' ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 2597 0 --:--:-- --:--:-- --:--:-- 2612 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-06 20:04:18.506","puller_resolved_ts":"2024-05-06 20:04:12.506","last_synced_ts":"2024-05-06 20:04:12.606","now_ts":"2024-05-06 20:04:19.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '20:04:18.506","puller_resolved_ts":"2024-05-06' '20:04:12.506","last_synced_ts":"2024-05-06' '20:04:12.606","now_ts":"2024-05-06' '20:04:19.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '20:04:18.506","puller_resolved_ts":"2024-05-06' '20:04:12.506","last_synced_ts":"2024-05-06' '20:04:12.606","now_ts":"2024-05-06' '20:04:19.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2571 0 --:--:-- --:--:-- --:--:-- 2600 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-06 20:06:28.656","puller_resolved_ts":"2024-05-06 20:06:21.657","last_synced_ts":"2024-05-06 20:04:12.606","now_ts":"2024-05-06 20:06:29.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '20:06:28.656","puller_resolved_ts":"2024-05-06' '20:06:21.657","last_synced_ts":"2024-05-06' '20:04:12.606","now_ts":"2024-05-06' '20:06:29.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '20:06:28.656","puller_resolved_ts":"2024-05-06' '20:06:21.657","last_synced_ts":"2024-05-06' '20:04:12.606","now_ts":"2024-05-06' '20:06:29.000","info":"Data' syncing is 'finished"}' ++ jq -r .info + info='Data syncing is finished' + target_message='Data syncing is finished' + '[' 'Data syncing is finished' '!=' 'Data syncing is finished' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_failpoint conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d373302680015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz, pid:25957, start at 2024-05-06 20:06:55.428462998 +0800 CST m=+5.154633033 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:08:55.438 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-20:06:55.436 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:56:55.436 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d373302680015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz, pid:25957, start at 2024-05-06 20:06:55.428462998 +0800 CST m=+5.154633033 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:08:55.438 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-20:06:55.436 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:56:55.436 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d373304a80009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:p-tiflow-release-8-1-pull-cdc-integration-storage-test-61-3c4dz, pid:26040, start at 2024-05-06 20:06:55.53862385 +0800 CST m=+5.199645455 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:08:55.547 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-20:06:55.530 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:56:55.530 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status + export 'GO_FAILPOINTS=github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.27409.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449576230947913729 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576230947913729 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449576230947913729 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Mon May 6 20:07:00 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2744827450.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 12:07:03 GMT < Content-Length: 829 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e1ea898b-b5e9-4f03-abbb-d09096035d66 {"id":"e1ea898b-b5e9-4f03-abbb-d09096035d66","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714997220} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dcc98bf34 e1ea898b-b5e9-4f03-abbb-d09096035d66 /tidb/cdc/default/default/upstream/7365856916987422028 {"id":7365856916987422028,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e1ea898b-b5e9-4f03-abbb-d09096035d66 {"id":"e1ea898b-b5e9-4f03-abbb-d09096035d66","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714997220} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dcc98bf34 e1ea898b-b5e9-4f03-abbb-d09096035d66 /tidb/cdc/default/default/upstream/7365856916987422028 {"id":7365856916987422028,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e1ea898b-b5e9-4f03-abbb-d09096035d66 {"id":"e1ea898b-b5e9-4f03-abbb-d09096035d66","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/bin/cdc.test","start-timestamp":1714997220} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dcc98bf34 e1ea898b-b5e9-4f03-abbb-d09096035d66 /tidb/cdc/default/default/upstream/7365856916987422028 {"id":7365856916987422028,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449576230947913729 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.27506.out cli changefeed create --start-ts=449576230947913729 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7365856916987422028,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-06T20:07:03.878641655+08:00","start_ts":449576230947913729,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576230947913729,"checkpoint_ts":449576230947913729,"checkpoint_time":"2024-05-06 20:06:58.887"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... + set +x + sleep 20 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 11305 0 --:--:-- --:--:-- --:--:-- 11296 100 723 100 723 0 0 11287 0 --:--:-- --:--:-- --:--:-- 11296 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-06 20:06:58.887","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-05-06 20:07:25.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '20:06:58.887","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-06' '20:07:25.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '20:06:58.887","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-06' '20:07:25.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + export GO_FAILPOINTS= + GO_FAILPOINTS= + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + check_logs /tmp/tidb_cdc_test/synced_status ++ date + echo '[Mon May 6 20:07:36 CST 2024] <<<<<< run test case synced_status success! >>>>>>' [Mon May 6 20:07:36 CST 2024] <<<<<< run test case synced_status success! >>>>>> + stop_tidb_cluster \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_storage_test-61/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // parallel [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] End of Pipeline Finished: SUCCESS