Started by user Jenkins Admin Obtained pipelines/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test.groovy from git https://github.com/PingCAP-QE/ci.git Loading library tipipeline@main Library tipipeline@main is cached. Copying from home. [Pipeline] Start of Pipeline [Pipeline] readJSON [Pipeline] readTrusted Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-6hj3q Agent ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-6hj3q is provisioned from template ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-k9pd0-j5r3k --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "fa47f17a9e5cde3c87acfe06a2d1344a57be90f8" jenkins/label: "ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-k9pd0" name: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-6hj3q" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:pulsar-test" name: "golang" resources: limits: memory: "16Gi" cpu: "8" requests: memory: "16Gi" cpu: "8" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-6hj3q" - name: "JENKINS_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-6hj3q" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-6hj3q in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout The recommended git tool is: git No credentials specified Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Commit message: "fix(br): use failpoint tidb-server instead (#2951)" First time build. Skipping changelog. [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 > git rev-list --no-walk a0a6bb5614d55afbab88f0d1ae459423175a5407 # timeout=10 [Pipeline] container [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] timeout Timeout set to expire in 1 hr 0 min [Pipeline] { [Pipeline] stage [Pipeline] { (Debug info) [Pipeline] sh + printenv PROW_JOB_ID=cd0f73e1-2b01-4d7c-b838-cad499e0f373 JENKINS_NODE_COOKIE=2f5f01d5-274b-446f-bfc8-ef017f826c56 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-6hj3q HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Debug info BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test","buildid":"1787445299229233158","prowjobid":"cd0f73e1-2b01-4d7c-b838-cad499e0f373","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/usr/java/openjdk-17/bin:/usr/local/pulsar/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct _=/usr/bin/printenv POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#72 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233158 GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=tests SHLVL=3 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-k9pd0 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-2ede9a5adcce94676487c5d3f2ac00b59960b70b812a72656e28f03ffa73fdb5 NODE_LABELS=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-k9pd0 ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-6hj3q GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-6hj3q GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=72 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz + echo ------------------------- ------------------------- + go env GO111MODULE='' GOARCH='amd64' GOBIN='' GOCACHE='/home/jenkins/.cache/go-build' GOENV='/home/jenkins/.config/go/env' GOEXE='' GOEXPERIMENT='' GOFLAGS='' GOHOSTARCH='amd64' GOHOSTOS='linux' GOINSECURE='' GOMODCACHE='/go/pkg/mod' GONOPROXY='' GONOSUMDB='' GOOS='linux' GOPATH='/go' GOPRIVATE='' GOPROXY='http://goproxy.apps.svc,https://proxy.golang.org,direct' GOROOT='/usr/local/go' GOSUMDB='sum.golang.org' GOTMPDIR='' GOTOOLCHAIN='auto' GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64' GOVCS='' GOVERSION='go1.21.6' GCCGO='gccgo' GOAMD64='v1' AR='ar' CC='gcc' CXX='g++' CGO_ENABLED='1' GOMOD='/dev/null' GOWORK='' CGO_CFLAGS='-O2 -g' CGO_CPPFLAGS='' CGO_CXXFLAGS='-O2 -g' CGO_FFLAGS='-O2 -g' CGO_LDFLAGS='-O2 -g' PKG_CONFIG='pkg-config' GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1677347432=/tmp/go-build -gno-record-gcc-switches' + echo ------------------------- ------------------------- + echo 'debug command: kubectl -n jenkins-tiflow exec -ti ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-6hj3q bash' debug command: kubectl -n jenkins-tiflow exec -ti ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-6hj3q bash [Pipeline] container [Pipeline] { [Pipeline] sh + dig github.com ; <<>> DiG 9.18.16 <<>> github.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37709 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ; COOKIE: 2cce294ea3e90ee6 (echoed) ;; QUESTION SECTION: ;github.com. IN A ;; ANSWER SECTION: github.com. 24 IN A 20.205.243.166 ;; Query time: 0 msec ;; SERVER: 169.254.25.10#53(169.254.25.10) (UDP) ;; WHEN: Mon May 06 11:32:58 UTC 2024 ;; MSG SIZE rcvd: 77 [Pipeline] script [Pipeline] { [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Check diff files) [Pipeline] container [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $token [Pipeline] { [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/11026/files?page=1&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/11026/files?page=1&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/11026/files?page=2&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/11026/files?page=2&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] } [Pipeline] // withCredentials [Pipeline] echo pr_diff_files: [examples/java/pom.xml] [Pipeline] echo diff file not matched: examples/java/pom.xml [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Checkout) [Pipeline] timeout Timeout set to expire in 10 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] cache Cache restored successfully (git/pingcap/tiflow/rev-936b4d4) 203908608 bytes in 1.11 secs (182920277 bytes/sec) [Pipeline] { [Pipeline] retry [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] sh git version 2.36.6 Reinitialized existing Git repository in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/.git/ .git HEAD is now at 936b4d4d2 kafka(ticdc): Fix java example license issue (#11024) POST git-upload-pack (686 bytes) POST git-upload-pack (1023 bytes) POST git-upload-pack (gzip 2173 to 1144 bytes) From https://github.com/pingcap/tiflow 3582359ab..38eb04081 release-8.1 -> origin/release-8.1 * [new ref] refs/pull/11026/head -> origin/pr/11026/head Previous HEAD position was 936b4d4d2 kafka(ticdc): Fix java example license issue (#11024) HEAD is now at 38eb04081 *(ticdc): Initialize changefeed/processor asynchronously (#10832) (#10985) ๐Ÿšง Checkouting to base SHA:38eb040812f817df410abe870952629df60fd01f... HEAD is now at 38eb04081 *(ticdc): Initialize changefeed/processor asynchronously (#10832) (#10985) โœ… Checked. ๐ŸŽ‰ ๐Ÿงพ HEAD info: 38eb040812f817df410abe870952629df60fd01f 38eb04081 *(ticdc): Initialize changefeed/processor asynchronously (#10832) (#10985) c78adb6f1 codec(ticdc): add more logs to the avro encoding to help detect potential key encoding failed issure (#10901) (#11002) 76cd91300 cli(ticdc): allow client authentication to be enabled without tls (#11005) (#11016) ๐Ÿšง Pre-merge heads of pull requests to base SHA: 38eb040812f817df410abe870952629df60fd01f ... Updating 38eb04081..e1a27adc6 Fast-forward examples/java/pom.xml | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) ๐Ÿงพ Pre-merged result: e1a27adc69acc9d50db0c3fea2945bbd6d983e30 e1a27adc6 fix java example license issue 38eb04081 *(ticdc): Initialize changefeed/processor asynchronously (#10832) (#10985) c78adb6f1 codec(ticdc): add more logs to the avro encoding to help detect potential key encoding failed issure (#10901) (#11002) โœ… Pre merged ๐ŸŽ‰ โœ… ~~~~~All done.~~~~~~ [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // retry [Pipeline] } Cache saved successfully (git/pingcap/tiflow/rev-38eb040-e1a27ad) 204049920 bytes in 3.94 secs (51848260 bytes/sec) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (prepare) [Pipeline] timeout Timeout set to expire in 20 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/third_party_download [Pipeline] { [Pipeline] retry [Pipeline] { [Pipeline] sh + cd ../tiflow + ./scripts/download-integration-test-binaries.sh release-8.1 Download binaries... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 41 100 41 0 0 736 0 --:--:-- --:--:-- --:--:-- 745 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 41 100 41 0 0 837 0 --:--:-- --:--:-- --:--:-- 854 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 41 100 41 0 0 894 0 --:--:-- --:--:-- --:--:-- 911 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 41 100 41 0 0 2418 0 --:--:-- --:--:-- --:--:-- 2562 >>> download tidb-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tidb/c8e6662d10deed030ef864169ae27718f5a61991/centos7/tidb-server.tar.gz 2024-05-06 19:33:27 URL:http://fileserver.pingcap.net/download/builds/pingcap/tidb/c8e6662d10deed030ef864169ae27718f5a61991/centos7/tidb-server.tar.gz [526223028/526223028] -> "tmp/tidb-server.tar.gz" [1] >>> download pd-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/pd/13eaf8aaa661dc0d3885ac9e59145f69752cee2d/centos7/pd-server.tar.gz 2024-05-06 19:33:44 URL:http://fileserver.pingcap.net/download/builds/pingcap/pd/13eaf8aaa661dc0d3885ac9e59145f69752cee2d/centos7/pd-server.tar.gz [184187328/184187328] -> "tmp/pd-server.tar.gz" [1] >>> download tikv-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tikv/56613f7c3e28c02853cc51d15bc1b77f68b58be8/centos7/tikv-server.tar.gz 2024-05-06 19:34:24 URL:http://fileserver.pingcap.net/download/builds/pingcap/tikv/56613f7c3e28c02853cc51d15bc1b77f68b58be8/centos7/tikv-server.tar.gz [834867730/834867730] -> "tmp/tikv-server.tar.gz" [1] >>> download tiflash.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tiflash/release-8.1/eb96e720183ea798458afc99cfb034864ed75c4c/centos7/tiflash.tar.gz 2024-05-06 19:34:55 URL:http://fileserver.pingcap.net/download/builds/pingcap/tiflash/release-8.1/eb96e720183ea798458afc99cfb034864ed75c4c/centos7/tiflash.tar.gz [455317481/455317481] -> "tmp/tiflash.tar.gz" [1] >>> download minio.tar.gz from http://fileserver.pingcap.net/download/minio.tar.gz 2024-05-06 19:35:01 URL:http://fileserver.pingcap.net/download/minio.tar.gz [17718777/17718777] -> "tmp/minio.tar.gz" [1] >>> download go-ycsb from http://fileserver.pingcap.net/download/builds/pingcap/go-ycsb/test-br/go-ycsb 2024-05-06 19:35:04 URL:http://fileserver.pingcap.net/download/builds/pingcap/go-ycsb/test-br/go-ycsb [45975512/45975512] -> "third_bin/go-ycsb" [1] >>> download jq from http://fileserver.pingcap.net/download/builds/pingcap/test/jq-1.6/jq-linux64 2024-05-06 19:35:04 URL:http://fileserver.pingcap.net/download/builds/pingcap/test/jq-1.6/jq-linux64 [3953824/3953824] -> "third_bin/jq" [1] >>> download etcd.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/etcd-v3.4.7-linux-amd64.tar.gz 2024-05-06 19:35:05 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/etcd-v3.4.7-linux-amd64.tar.gz [17310840/17310840] -> "tmp/etcd.tar.gz" [1] >>> download sync_diff_inspector.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/sync_diff_inspector_hash-d671b084_linux-amd64.tar.gz 2024-05-06 19:35:10 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/sync_diff_inspector_hash-d671b084_linux-amd64.tar.gz [79877126/79877126] -> "tmp/sync_diff_inspector.tar.gz" [1] >>> download schema-registry.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/schema-registry.tar.gz 2024-05-06 19:35:23 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/schema-registry.tar.gz [278386006/278386006] -> "tmp/schema-registry.tar.gz" [1] Download SUCCESS + ls -alh ./bin total 1.9G drwxr-sr-x 6 jenkins jenkins 4.0K May 6 19:35 . drwxr-sr-x 19 jenkins jenkins 4.0K May 6 19:35 .. drwxr-sr-x 2 jenkins jenkins 4.0K May 19 2023 bin drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 etc -rwxr-xr-x 1 jenkins jenkins 17M Apr 2 2020 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M May 6 19:35 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M May 6 19:35 jq drwxr-sr-x 3 jenkins jenkins 4.0K May 10 2023 lib lrwxrwxrwx 1 jenkins jenkins 13 May 5 14:57 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1016K Nov 7 01:00 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 May 5 14:57 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 358K Nov 7 01:00 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 May 5 14:57 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 May 5 14:57 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.6M May 5 14:18 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 272M May 5 14:58 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M Jul 29 2020 minio -rwxr-xr-x 1 jenkins jenkins 37M Apr 30 15:24 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 44M Apr 30 15:24 pd-ctl -rwxr-xr-x 1 jenkins jenkins 36M Apr 30 15:24 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 32M Apr 30 15:24 pd-recover -rwxr-xr-x 1 jenkins jenkins 106M Apr 30 15:23 pd-server -rwxr-xr-x 1 jenkins jenkins 26M Apr 30 15:24 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 32M Apr 30 15:24 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 share -rwxr-xr-x 1 jenkins jenkins 32M Apr 30 15:24 stores-dump -rwxr-xr-x 1 jenkins jenkins 192M Sep 22 2023 sync_diff_inspector -rwxr-xr-x 1 jenkins jenkins 205M May 6 18:40 tidb-server -rwxr-xr-x 1 jenkins jenkins 379M May 5 14:58 tiflash -rwxr-xr-x 1 jenkins jenkins 367M Apr 30 15:02 tikv-server + make check_third_party_binary /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/tidb-server /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/tikv-server /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/pd-server /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/tiflash /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/pd-ctl /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/sync_diff_inspector /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/go-ycsb /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/etcdctl /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/jq /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/minio /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/bin/schema-registry-start + cd - /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/third_party_download + mkdir -p bin + mv ../tiflow/bin/bin ../tiflow/bin/etc ../tiflow/bin/etcdctl ../tiflow/bin/go-ycsb ../tiflow/bin/jq ../tiflow/bin/lib ../tiflow/bin/libc++.so.1 ../tiflow/bin/libc++.so.1.0 ../tiflow/bin/libc++abi.so.1 ../tiflow/bin/libc++abi.so.1.0 ../tiflow/bin/libgmssl.so ../tiflow/bin/libgmssl.so.3 ../tiflow/bin/libgmssl.so.3.0 ../tiflow/bin/libtiflash_proxy.so ../tiflow/bin/minio ../tiflow/bin/pd-api-bench ../tiflow/bin/pd-ctl ../tiflow/bin/pd-heartbeat-bench ../tiflow/bin/pd-recover ../tiflow/bin/pd-server ../tiflow/bin/pd-tso-bench ../tiflow/bin/regions-dump ../tiflow/bin/share ../tiflow/bin/stores-dump ../tiflow/bin/sync_diff_inspector ../tiflow/bin/tidb-server ../tiflow/bin/tiflash ../tiflow/bin/tikv-server ./bin/ + ls -alh ./bin total 1.9G drwxr-sr-x 6 jenkins jenkins 4.0K May 6 19:35 . drwxr-sr-x 3 jenkins jenkins 4.0K May 6 19:35 .. drwxr-sr-x 2 jenkins jenkins 4.0K May 19 2023 bin drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 etc -rwxr-xr-x 1 jenkins jenkins 17M Apr 2 2020 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M May 6 19:35 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M May 6 19:35 jq drwxr-sr-x 3 jenkins jenkins 4.0K May 10 2023 lib lrwxrwxrwx 1 jenkins jenkins 13 May 5 14:57 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1016K Nov 7 01:00 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 May 5 14:57 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 358K Nov 7 01:00 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 May 5 14:57 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 May 5 14:57 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.6M May 5 14:18 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 272M May 5 14:58 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M Jul 29 2020 minio -rwxr-xr-x 1 jenkins jenkins 37M Apr 30 15:24 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 44M Apr 30 15:24 pd-ctl -rwxr-xr-x 1 jenkins jenkins 36M Apr 30 15:24 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 32M Apr 30 15:24 pd-recover -rwxr-xr-x 1 jenkins jenkins 106M Apr 30 15:23 pd-server -rwxr-xr-x 1 jenkins jenkins 26M Apr 30 15:24 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 32M Apr 30 15:24 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 share -rwxr-xr-x 1 jenkins jenkins 32M Apr 30 15:24 stores-dump -rwxr-xr-x 1 jenkins jenkins 192M Sep 22 2023 sync_diff_inspector -rwxr-xr-x 1 jenkins jenkins 205M May 6 18:40 tidb-server -rwxr-xr-x 1 jenkins jenkins 379M May 5 14:58 tiflash -rwxr-xr-x 1 jenkins jenkins 367M Apr 30 15:02 tikv-server + ./bin/tidb-server -V Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore + ./bin/pd-server -V Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 + ./bin/tikv-server -V TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + ./bin/tiflash --version TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored + ./bin/sync_diff_inspector --version App Name: sync_diff_inspector v2.0 Release Version: v7.4.0 Git Commit Hash: d671b0840063bc2532941f02e02e12627402844c Git Branch: heads/refs/tags/v7.4.0 UTC Build Time: 2023-09-22 03:51:56 Go Version: go1.21.1 [Pipeline] } [Pipeline] // retry [Pipeline] } [Pipeline] // dir [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] cache Cache not restored (no such key found) [Pipeline] { [Pipeline] sh + ls -alh ./bin total 8.0K drwxr-sr-x 2 jenkins jenkins 4.0K May 6 19:35 . drwxr-sr-x 19 jenkins jenkins 4.0K May 6 19:35 .. + '[' -f ./bin/cdc ']' + make cdc CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v8.1.0-alpha-23-ge1a27adc6" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-06 11:35:27" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e1a27adc69acc9d50db0c3fea2945bbd6d983e30" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v8.1.0-alpha-23-ge1a27adc6"' -o bin/cdc ./cmd/cdc go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20240415145106-cd9c676e9ba4 go: downloading google.golang.org/grpc v1.62.1 go: downloading github.com/pingcap/failpoint v0.0.0-20220801062533-2eaa32854a6c go: downloading github.com/spf13/cobra v1.8.0 go: downloading github.com/IBM/sarama v1.41.2 go: downloading github.com/fatih/color v1.16.0 go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/tikv/pd/client v0.0.0-20240322051414-fb9e2d561b6e go: downloading golang.org/x/net v0.24.0 go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20240410110152-5fc42c9be2f5 go: downloading github.com/xdg/scram v1.0.5 go: downloading golang.org/x/sync v0.7.0 go: downloading github.com/gin-gonic/gin v1.9.1 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/BurntSushi/toml v1.3.2 go: downloading github.com/prometheus/client_golang v1.19.0 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/pingcap/kvproto v0.0.0-20240227073058-929ab83f9754 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20240409022718-714958ccd4d5 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.1 go: downloading github.com/KimMachineGun/automemlimit v0.2.4 go: downloading github.com/aws/aws-sdk-go v1.50.0 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/shirou/gopsutil/v3 v3.24.2 go: downloading github.com/r3labs/diff v1.1.0 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/DATA-DOG/go-sqlmock v1.5.0 go: downloading github.com/imdario/mergo v0.3.16 go: downloading github.com/google/btree v1.1.2 go: downloading github.com/apache/pulsar-client-go v0.11.0 go: downloading github.com/aws/aws-sdk-go-v2 v1.19.1 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading github.com/pingcap/tidb-tools v0.0.0-20240305021104-9f9bea84490b go: downloading github.com/tinylib/msgp v1.1.6 go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading go.uber.org/atomic v1.11.0 go: downloading github.com/swaggo/files v0.0.0-20210815190702-a29dd2bc99b2 go: downloading github.com/swaggo/gin-swagger v1.2.0 go: downloading github.com/google/uuid v1.6.0 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading go.etcd.io/etcd/server/v3 v3.5.12 go: downloading golang.org/x/time v0.5.0 go: downloading cloud.google.com/go v0.112.2 go: downloading github.com/stretchr/testify v1.9.0 go: downloading github.com/cenkalti/backoff/v4 v4.2.1 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/klauspost/compress v1.17.8 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/uber-go/atomic v1.4.0 go: downloading go.etcd.io/etcd/pkg/v3 v3.5.12 go: downloading github.com/soheilhy/cmux v0.1.5 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/goccy/go-json v0.10.2 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading github.com/xdg/stringprep v1.0.3 go: downloading golang.org/x/crypto v0.22.0 go: downloading golang.org/x/sys v0.19.0 go: downloading github.com/YangKeao/seahash v0.0.0-20240229041150-e7bf269c3140 go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading github.com/benbjohnson/clock v1.3.5 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading github.com/hashicorp/golang-lru v0.5.1 go: downloading github.com/jcmturner/gokrb5/v8 v8.4.4 go: downloading github.com/segmentio/kafka-go v0.4.41-0.20230526171612-f057b1d369cd go: downloading github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 go: downloading golang.org/x/oauth2 v0.18.0 go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2 go: downloading github.com/gavv/monotime v0.0.0-20190418164738-30dba4353424 go: downloading golang.org/x/term v0.19.0 go: downloading github.com/containerd/cgroups v1.0.4 go: downloading github.com/golang/mock v1.6.0 go: downloading github.com/go-mysql-org/go-mysql v1.7.1-0.20240314115043-2199dfb0ba98 go: downloading github.com/pingcap/check v0.0.0-20211026125417-57bd13f7b5f0 go: downloading github.com/gin-contrib/sse v0.1.0 go: downloading github.com/go-playground/validator/v10 v10.14.0 go: downloading github.com/pelletier/go-toml/v2 v2.0.8 go: downloading github.com/ugorji/go/codec v1.2.11 go: downloading google.golang.org/protobuf v1.33.0 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/swaggo/swag v1.16.3 go: downloading github.com/philhofer/fwd v1.1.1 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/prometheus/common v0.52.2 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/prometheus/procfs v0.13.0 go: downloading github.com/pingcap/tidb-dashboard v0.0.0-20240326110213-9768844ff5d7 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/eapache/go-resiliency v1.4.0 go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 go: downloading github.com/eapache/queue v1.1.0 go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/jcmturner/gofork v1.7.6 go: downloading github.com/aws/smithy-go v1.13.5 go: downloading github.com/golang/protobuf v1.5.4 go: downloading cloud.google.com/go/compute/metadata v0.2.3 go: downloading cloud.google.com/go/compute v1.25.1 go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading google.golang.org/api v0.170.0 go: downloading github.com/pierrec/lz4 v2.6.1+incompatible go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading github.com/bits-and-blooms/bitset v1.4.0 go: downloading github.com/linkedin/goavro/v2 v2.11.1 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading github.com/stretchr/objx v0.5.2 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading gorm.io/gorm v1.24.5 go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading golang.org/x/text v0.14.0 go: downloading github.com/cakturk/go-netstat v0.0.0-20200220111822-e5b49efee7a5 go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0 go: downloading go.etcd.io/bbolt v1.3.9 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading go.opentelemetry.io/otel/sdk v1.22.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.22.0 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading github.com/AthenZ/athenz v1.10.39 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240401170217-c3f982113cda go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/jcmturner/dnsutils/v2 v2.0.0 go: downloading github.com/hashicorp/go-uuid v1.0.3 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/godbus/dbus/v5 v5.0.4 go: downloading github.com/cilium/ebpf v0.4.0 go: downloading github.com/opencontainers/runtime-spec v1.0.2 go: downloading github.com/xdg-go/scram v1.1.2 go: downloading github.com/Masterminds/semver v1.5.0 go: downloading github.com/siddontang/go-log v0.0.0-20180807004314-8d05993dda07 go: downloading github.com/siddontang/go v0.0.0-20180604090527-bdc77568d726 go: downloading github.com/gabriel-vasile/mimetype v1.4.2 go: downloading github.com/go-playground/universal-translator v0.18.1 go: downloading github.com/leodido/go-urn v1.2.4 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 go: downloading golang.org/x/exp v0.0.0-20240409090435-93d18d7e34b8 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.1 go: downloading github.com/KyleBanks/depth v1.2.1 go: downloading github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 go: downloading github.com/go-openapi/spec v0.21.0 go: downloading golang.org/x/tools v0.20.0 go: downloading github.com/joomcode/errorx v1.0.1 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/hashicorp/errwrap v1.0.0 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.1 go: downloading github.com/spaolacci/murmur3 v1.1.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading golang.org/x/mod v0.17.0 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/otiai10/copy v1.2.0 go: downloading github.com/gorilla/mux v1.8.0 go: downloading github.com/blacktear23/go-proxyprotocol v1.0.6 go: downloading github.com/pingcap/fn v1.0.0 go: downloading github.com/pingcap/tipb v0.0.0-20240318032315-55a7867ddd50 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading github.com/glebarez/sqlite v1.7.0 go: downloading github.com/scalalang2/golang-fifo v0.1.5 go: downloading github.com/tidwall/btree v1.7.0 go: downloading github.com/jonboulle/clockwork v0.4.0 go: downloading github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 go: downloading go.etcd.io/etcd/raft/v3 v3.5.12 go: downloading gorm.io/driver/mysql v1.3.3 go: downloading github.com/golang-jwt/jwt v3.2.2+incompatible go: downloading github.com/99designs/keyring v1.2.1 go: downloading github.com/jcmturner/aescts/v2 v2.0.0 go: downloading github.com/jcmturner/rpc/v2 v2.0.3 go: downloading github.com/tiancaiamao/appdash v0.0.0-20181126055449-889f96f722a2 go: downloading github.com/sourcegraph/appdash-data v0.0.0-20151005221446-73f23eafcf67 go: downloading github.com/aws/aws-sdk-go-v2/config v1.18.30 go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.13.29 go: downloading github.com/aws/aws-sdk-go-v2/service/glue v1.58.1 go: downloading github.com/jarcoal/httpmock v1.2.0 go: downloading github.com/mailru/easyjson v0.7.7 go: downloading github.com/shopspring/decimal v1.3.0 go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/gorilla/websocket v1.5.1 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading github.com/xdg-go/pbkdf2 v1.0.0 go: downloading github.com/xdg-go/stringprep v1.0.4 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/go-playground/locales v0.14.1 go: downloading github.com/jinzhu/now v1.1.5 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.1 go: downloading github.com/go-openapi/jsonpointer v0.21.0 go: downloading github.com/go-openapi/jsonreference v0.21.0 go: downloading github.com/go-openapi/swag v0.23.0 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/danjacques/gofslock v0.0.0-20240212154529-d899e02bfe22 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/go-logr/logr v1.4.1 go: downloading go.opentelemetry.io/proto/otlp v1.1.0 go: downloading github.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0 go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading github.com/glebarez/go-sqlite v1.21.2 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/go-ozzo/ozzo-validation/v4 v4.3.0 go: downloading github.com/golang-jwt/jwt/v4 v4.5.0 go: downloading github.com/edwingeng/deque v0.0.0-20191220032131-8596380dee17 go: downloading go.etcd.io/etcd/client/v2 v2.305.12 go: downloading github.com/google/pprof v0.0.0-20240117000934-35fc243c5815 go: downloading github.com/dvsekhvalnov/jose2go v1.5.0 go: downloading github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c go: downloading github.com/mtibben/percent v0.2.1 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.0 go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.6 go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.3.37 go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.12.14 go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.14 go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.20.1 go: downloading github.com/jinzhu/inflection v1.0.0 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.36 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/mattn/go-runewidth v0.0.15 go: downloading github.com/pingcap/badger v1.5.1-0.20230103063557-828f39b09b6d go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 go: downloading github.com/vbauerster/mpb/v7 v7.5.3 go: downloading github.com/jfcg/sixb v1.3.8 go: downloading modernc.org/libc v1.37.1 go: downloading modernc.org/sqlite v1.27.0 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.30 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.30 go: downloading github.com/golang-jwt/jwt/v5 v5.2.0 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.12.0 go: downloading github.com/josharian/intern v1.0.0 go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d go: downloading k8s.io/api v0.28.6 go: downloading github.com/emirpasic/gods v1.18.1 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading k8s.io/apimachinery v0.28.6 go: downloading github.com/ardielle/ardielle-go v1.5.2 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading k8s.io/klog/v2 v2.120.1 go: downloading github.com/google/gofuzz v1.2.0 go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd go: downloading github.com/jmespath/go-jmespath v0.4.0 go: downloading modernc.org/memory v1.7.2 go: downloading modernc.org/mathutil v1.6.0 go: downloading github.com/google/s2a-go v0.1.7 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading github.com/felixge/httpsnoop v1.0.4 + '[' -f ./bin/cdc_pulsar_consumer ']' + make pulsar_consumer CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v8.1.0-alpha-23-ge1a27adc6" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-06 11:37:52" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e1a27adc69acc9d50db0c3fea2945bbd6d983e30" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v8.1.0-alpha-23-ge1a27adc6"' -o bin/cdc_pulsar_consumer ./cmd/pulsar-consumer/main.go + '[' -f ./bin/cdc.test ']' + make integration_test_build cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/failpoint-ctl github.com/pingcap/failpoint/failpoint-ctl go: downloading github.com/pingcap/failpoint v0.0.0-20210316064728-7acb0f0a3dfd go: downloading github.com/sergi/go-diff v1.1.0 CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v8.1.0-alpha-23-ge1a27adc6" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-06 11:38:00" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e1a27adc69acc9d50db0c3fea2945bbd6d983e30" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v8.1.0-alpha-23-ge1a27adc6"' -o bin/cdc_storage_consumer ./cmd/storage-consumer/main.go CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v8.1.0-alpha-23-ge1a27adc6" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-06 11:38:00" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e1a27adc69acc9d50db0c3fea2945bbd6d983e30" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v8.1.0-alpha-23-ge1a27adc6"' -o bin/cdc_kafka_consumer ./cmd/kafka-consumer/main.go CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v8.1.0-alpha-23-ge1a27adc6" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-06 11:38:00" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e1a27adc69acc9d50db0c3fea2945bbd6d983e30" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v8.1.0-alpha-23-ge1a27adc6"' -o bin/cdc_pulsar_consumer ./cmd/pulsar-consumer/main.go CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v8.1.0-alpha-23-ge1a27adc6" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-06 11:38:00" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e1a27adc69acc9d50db0c3fea2945bbd6d983e30" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v8.1.0-alpha-23-ge1a27adc6"' -o bin/oauth2-server ./cmd/oauth2-server/main.go go: downloading github.com/go-oauth2/oauth2/v4 v4.5.2 go: downloading github.com/tidwall/buntdb v1.3.0 go: downloading github.com/tidwall/rtred v0.1.2 go: downloading github.com/tidwall/grect v0.1.4 go: downloading github.com/tidwall/match v1.1.1 go: downloading github.com/tidwall/gjson v1.14.3 go: downloading github.com/tidwall/tinyqueue v0.1.1 go: downloading github.com/tidwall/pretty v1.2.0 $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl enable >/dev/null) go: downloading github.com/PingCAP-QE/go-sqlsmith v0.0.0-20231213065948-336e064b488d go: downloading github.com/chzyer/readline v1.5.1 go: downloading github.com/deepmap/oapi-codegen v1.9.0 go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 go: downloading github.com/gogo/gateway v1.1.0 go: downloading github.com/getkin/kin-openapi v0.80.0 go: downloading github.com/syndtr/goleveldb v1.0.1-0.20210305035536-64b5b1c73954 go: downloading go.uber.org/dig v1.13.0 go: downloading github.com/mattn/go-shellwords v1.0.12 go: downloading go.uber.org/ratelimit v0.2.0 go: downloading github.com/VividCortex/mysqlerr v1.0.0 go: downloading go.uber.org/goleak v1.3.0 go: downloading github.com/bradleyjkemp/grpc-tools v0.2.5 go: downloading github.com/integralist/go-findroot v0.0.0-20160518114804-ac90681525dc go: downloading github.com/jmoiron/sqlx v1.3.3 go: downloading upper.io/db.v3 v3.7.1+incompatible go: downloading github.com/DataDog/zstd v1.5.5 go: downloading github.com/shurcooL/httpgzip v0.0.0-20190720172056-320755c1c1b0 go: downloading github.com/andres-erbsen/clock v0.0.0-20160526145045-9e14626cd129 go: downloading github.com/ngaut/log v0.0.0-20210830112240-0124ec040aeb go: downloading github.com/improbable-eng/grpc-web v0.12.0 go: downloading github.com/ghodss/yaml v1.0.0 go: downloading github.com/rs/cors v1.7.0 go: downloading github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f CGO_ENABLED=1 GO111MODULE=on go test -p 3 --race --tags=intest -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v8.1.0-alpha-23-ge1a27adc6" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-06 11:38:00" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e1a27adc69acc9d50db0c3fea2945bbd6d983e30" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v8.1.0-alpha-23-ge1a27adc6"' -c -cover -covermode=atomic \ -coverpkg=github.com/pingcap/tiflow/... \ -o bin/cdc.test github.com/pingcap/tiflow/cmd/cdc \ || { $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null); exit 1; } CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v8.1.0-alpha-23-ge1a27adc6" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-05-06 11:38:00" -X "github.com/pingcap/tiflow/pkg/version.GitHash=e1a27adc69acc9d50db0c3fea2945bbd6d983e30" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v8.1.0-alpha-23-ge1a27adc6"' -o bin/cdc ./cmd/cdc/main.go \ || { $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null); exit 1; } $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null) + ls -alh ./bin total 1.2G drwxr-sr-x 2 jenkins jenkins 4.0K May 6 19:42 . drwxr-sr-x 19 jenkins jenkins 4.0K May 6 19:35 .. -rwxr-xr-x 1 jenkins jenkins 220M May 6 19:42 cdc -rwxr-xr-x 1 jenkins jenkins 363M May 6 19:42 cdc.test -rwxr-xr-x 1 jenkins jenkins 183M May 6 19:38 cdc_kafka_consumer -rwxr-xr-x 1 jenkins jenkins 182M May 6 19:38 cdc_pulsar_consumer -rwxr-xr-x 1 jenkins jenkins 182M May 6 19:38 cdc_storage_consumer -rwxr-xr-x 1 jenkins jenkins 12M May 6 19:38 oauth2-server + ./bin/cdc version Release Version: v8.1.0-alpha-23-ge1a27adc6 Git Commit Hash: e1a27adc69acc9d50db0c3fea2945bbd6d983e30 Git Branch: HEAD UTC Build Time: 2024-05-06 11:38:00 Go Version: go version go1.21.6 linux/amd64 Failpoint Build: true [Pipeline] } Cache saved successfully (binary/pingcap/tiflow/cdc-integration-pulsar-test/rev-38eb040-e1a27ad) 1194937344 bytes in 17.09 secs (69934207 bytes/sec) [Pipeline] // cache [Pipeline] cache Cache not restored (no such key found) [Pipeline] { [Pipeline] sh + cp -r ../third_party_download/bin/bin ../third_party_download/bin/etc ../third_party_download/bin/etcdctl ../third_party_download/bin/go-ycsb ../third_party_download/bin/jq ../third_party_download/bin/lib ../third_party_download/bin/libc++.so.1 ../third_party_download/bin/libc++.so.1.0 ../third_party_download/bin/libc++abi.so.1 ../third_party_download/bin/libc++abi.so.1.0 ../third_party_download/bin/libgmssl.so ../third_party_download/bin/libgmssl.so.3 ../third_party_download/bin/libgmssl.so.3.0 ../third_party_download/bin/libtiflash_proxy.so ../third_party_download/bin/minio ../third_party_download/bin/pd-api-bench ../third_party_download/bin/pd-ctl ../third_party_download/bin/pd-heartbeat-bench ../third_party_download/bin/pd-recover ../third_party_download/bin/pd-server ../third_party_download/bin/pd-tso-bench ../third_party_download/bin/regions-dump ../third_party_download/bin/share ../third_party_download/bin/stores-dump ../third_party_download/bin/sync_diff_inspector ../third_party_download/bin/tidb-server ../third_party_download/bin/tiflash ../third_party_download/bin/tikv-server ./bin/ + ls -alh ./bin total 3.0G drwxr-sr-x 6 jenkins jenkins 4.0K May 6 19:43 . drwxr-sr-x 19 jenkins jenkins 4.0K May 6 19:35 .. drwxr-sr-x 2 jenkins jenkins 4.0K May 6 19:43 bin -rwxr-xr-x 1 jenkins jenkins 220M May 6 19:42 cdc -rwxr-xr-x 1 jenkins jenkins 363M May 6 19:42 cdc.test -rwxr-xr-x 1 jenkins jenkins 183M May 6 19:38 cdc_kafka_consumer -rwxr-xr-x 1 jenkins jenkins 182M May 6 19:38 cdc_pulsar_consumer -rwxr-xr-x 1 jenkins jenkins 182M May 6 19:38 cdc_storage_consumer drwxr-sr-x 4 jenkins jenkins 4.0K May 6 19:43 etc -rwxr-xr-x 1 jenkins jenkins 17M May 6 19:43 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M May 6 19:43 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M May 6 19:43 jq drwxr-sr-x 3 jenkins jenkins 4.0K May 6 19:43 lib lrwxrwxrwx 1 jenkins jenkins 13 May 6 19:43 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1016K May 6 19:43 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 May 6 19:43 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 358K May 6 19:43 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 May 6 19:43 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 May 6 19:43 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.6M May 6 19:43 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 272M May 6 19:43 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M May 6 19:43 minio -rwxr-xr-x 1 jenkins jenkins 12M May 6 19:38 oauth2-server -rwxr-xr-x 1 jenkins jenkins 37M May 6 19:43 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 44M May 6 19:43 pd-ctl -rwxr-xr-x 1 jenkins jenkins 36M May 6 19:43 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 32M May 6 19:43 pd-recover -rwxr-xr-x 1 jenkins jenkins 106M May 6 19:43 pd-server -rwxr-xr-x 1 jenkins jenkins 26M May 6 19:43 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 32M May 6 19:43 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K May 6 19:43 share -rwxr-xr-x 1 jenkins jenkins 32M May 6 19:43 stores-dump -rwxr-xr-x 1 jenkins jenkins 192M May 6 19:43 sync_diff_inspector -rwxr-xr-x 1 jenkins jenkins 205M May 6 19:43 tidb-server -rwxr-xr-x 1 jenkins jenkins 379M May 6 19:43 tiflash -rwxr-xr-x 1 jenkins jenkins 367M May 6 19:43 tikv-server [Pipeline] } Cache saved successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc) 3667975680 bytes in 76.10 secs (48200423 bytes/sec) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Tests) [Pipeline] parallel [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G00') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G01') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G02') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G03') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G04') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G05') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G06') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G07') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G08') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G09') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G10') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G11') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G12') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G13') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G14') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G15') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G16') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G17') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G00') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G01') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G02') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G03') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G04') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G05') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G06') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G07') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G08') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G09') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G10') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G11') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G12') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G13') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G14') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G15') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G16') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G17') [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-t8cg2 Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-z1579 [Pipeline] podTemplate [Pipeline] { [Pipeline] node Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-k03d6 Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-55h44 Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-105q1 Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-b9q8w Agent ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-t8cg2 is provisioned from template ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-07t56-9xj93 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "03fbb3e2227a890bcf56802267cacd56e72e2361" jenkins/label: "ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-07t56" name: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-t8cg2" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:pulsar-test" name: "golang" resources: limits: memory: "16Gi" cpu: "8" requests: memory: "16Gi" cpu: "8" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-t8cg2" - name: "JENKINS_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-t8cg2" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-t8cg2 in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test [Pipeline] { [Pipeline] checkout The recommended git tool is: git Agent ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-z1579 is provisioned from template ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-m6xj7-1bpfv --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "309112c8eae9bcc2650cd7861ea412dbdc989599" jenkins/label: "ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-m6xj7" name: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-z1579" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:pulsar-test" name: "golang" resources: limits: memory: "16Gi" cpu: "8" requests: memory: "16Gi" cpu: "8" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-z1579" - name: "JENKINS_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-z1579" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-z1579 in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test [Pipeline] { [Pipeline] checkout The recommended git tool is: git Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jzh76 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@33e28a85; decorates RemoteLauncher[hudson.remoting.Channel@3d2c5dae:JNLP4-connect connection from 10.233.67.58/10.233.67.58:39664] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Agent ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-k03d6 is provisioned from template ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-sd01c-w0b34 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "7ec008be24b92bad85829892c6a4e7412d83c4a5" jenkins/label: "ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-sd01c" name: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-k03d6" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:pulsar-test" name: "golang" resources: limits: memory: "16Gi" cpu: "8" requests: memory: "16Gi" cpu: "8" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-k03d6" - name: "JENKINS_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-k03d6" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-k03d6 in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test [Pipeline] { No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@3d7a611a; decorates RemoteLauncher[hudson.remoting.Channel@3833050e:JNLP4-connect connection from 10.233.72.16/10.233.72.16:60288] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] checkout Agent ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-55h44 is provisioned from template ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-5hwc4-qbr4m --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "b92d69b84d6587cb8455f1f2d9c2b56755430687" jenkins/label: "ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-5hwc4" name: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-55h44" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:pulsar-test" name: "golang" resources: limits: memory: "16Gi" cpu: "8" requests: memory: "16Gi" cpu: "8" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-55h44" - name: "JENKINS_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-55h44" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" The recommended git tool is: git Running on ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-55h44 in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] { Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-fqtgg [Pipeline] checkout The recommended git tool is: git Agent ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-105q1 is provisioned from template ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-pccmc-gn4k7 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "7c62831a42b9ee0797274678b087042bdd17d99c" jenkins/label: "ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-pccmc" name: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-105q1" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:pulsar-test" name: "golang" resources: limits: memory: "16Gi" cpu: "8" requests: memory: "16Gi" cpu: "8" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-105q1" - name: "JENKINS_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-105q1" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-105q1 in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test [Pipeline] { [Pipeline] checkout The recommended git tool is: git No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@4dd7fd7e; decorates RemoteLauncher[hudson.remoting.Channel@cec6383:JNLP4-connect connection from 10.233.70.96/10.233.70.96:36348] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@77653aa8; decorates RemoteLauncher[hudson.remoting.Channel@4842235e:JNLP4-connect connection from 10.233.66.206/10.233.66.206:56514] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@414c18d6; decorates RemoteLauncher[hudson.remoting.Channel@37c60154:JNLP4-connect connection from 10.233.71.145/10.233.71.145:49976] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Agent ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jzh76 is provisioned from template ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-ppcz6-pfl85 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "28326d14dbc93aa76479615d0e1f684f2d5879fd" jenkins/label: "ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-ppcz6" name: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jzh76" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:pulsar-test" name: "golang" resources: limits: memory: "16Gi" cpu: "8" requests: memory: "16Gi" cpu: "8" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jzh76" - name: "JENKINS_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jzh76" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jzh76 in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test [Pipeline] { [Pipeline] checkout The recommended git tool is: git Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb Agent ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-b9q8w is provisioned from template ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-gdv1f-cns8x --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "da524c89f74d892c3ab760b422634911136eb83e" jenkins/label: "ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-gdv1f" name: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-b9q8w" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:pulsar-test" name: "golang" resources: limits: memory: "16Gi" cpu: "8" requests: memory: "16Gi" cpu: "8" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-b9q8w" - name: "JENKINS_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-b9q8w" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-b9q8w in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test [Pipeline] { [Pipeline] checkout The recommended git tool is: git Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@6d54f9de; decorates RemoteLauncher[hudson.remoting.Channel@159081f3:JNLP4-connect connection from 10.233.106.83/10.233.106.83:57644] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Commit message: "fix(br): use failpoint tidb-server instead (#2951)" [Pipeline] withEnv [Pipeline] { Agent ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-fqtgg is provisioned from template ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-lj75x-12k63 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "f36a3974f99427c4477a8ea0559db99a6f677448" jenkins/label: "ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-lj75x" name: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-fqtgg" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:pulsar-test" name: "golang" resources: limits: memory: "16Gi" cpu: "8" requests: memory: "16Gi" cpu: "8" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-fqtgg" - name: "JENKINS_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-fqtgg" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Commit message: "fix(br): use failpoint tidb-server instead (#2951)" Running on ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-fqtgg in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test [Pipeline] container [Pipeline] { [Pipeline] { [Pipeline] withEnv [Pipeline] { Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] checkout > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 [Pipeline] container [Pipeline] { Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 The recommended git tool is: git [Pipeline] podTemplate No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@5940d87b; decorates RemoteLauncher[hudson.remoting.Channel@44ace504:JNLP4-connect connection from 10.233.100.187/10.233.100.187:45454] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh [Pipeline] { [Pipeline] node [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] cache > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Agent ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6 is provisioned from template ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-spg9v-4d626 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "56f7df1b14f5d2a2e8f2eb08ce40e7a3006d6993" jenkins/label: "ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-spg9v" name: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:pulsar-test" name: "golang" resources: limits: memory: "16Gi" cpu: "8" requests: memory: "16Gi" cpu: "8" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6" - name: "JENKINS_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6 in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@3e55afa9; decorates RemoteLauncher[hudson.remoting.Channel@19798147:JNLP4-connect connection from 10.233.107.119/10.233.107.119:48088] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-v1jxz Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Commit message: "fix(br): use failpoint tidb-server instead (#2951)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb is provisioned from template ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-nxj9w-0ccs8 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "6517855af99ecbe9902ed006b464a2f5d5363dc7" jenkins/label: "ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-nxj9w" name: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:pulsar-test" name: "golang" resources: limits: memory: "16Gi" cpu: "8" requests: memory: "16Gi" cpu: "8" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb" - name: "JENKINS_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Commit message: "fix(br): use failpoint tidb-server instead (#2951)" > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 Commit message: "fix(br): use failpoint tidb-server instead (#2951)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Agent ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh is provisioned from template ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-7n0m1-5s0d5 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "c7d337f2b52a19fa93025c33c9b9b95b27127a36" jenkins/label: "ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-7n0m1" name: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:pulsar-test" name: "golang" resources: limits: memory: "16Gi" cpu: "8" requests: memory: "16Gi" cpu: "8" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh" - name: "JENKINS_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Commit message: "fix(br): use failpoint tidb-server instead (#2951)" Commit message: "fix(br): use failpoint tidb-server instead (#2951)" Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Agent ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-v1jxz is provisioned from template ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-4nhnt-d58r5 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "24ec20b5a350f2d8515378af4c6963f99cb55eff" jenkins/label: "ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-4nhnt" name: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-v1jxz" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:pulsar-test" name: "golang" resources: limits: memory: "16Gi" cpu: "8" requests: memory: "16Gi" cpu: "8" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-v1jxz" - name: "JENKINS_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-v1jxz" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-v1jxz in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test Commit message: "fix(br): use failpoint tidb-server instead (#2951)" > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 Obtained pipelines/pingcap/tiflow/release-8.1/pod-pull_cdc_integration_pulsar_test.yaml from git https://github.com/PingCAP-QE/ci.git Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc) 3667975680 bytes in 9.04 secs (405560469 bytes/sec) [Pipeline] { [Pipeline] sh [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] cache + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G01 Run cases: http_api http_api_tls api_v2 http_api_tls_with_user_auth cli_tls_with_auth kafka_simple_basic kafka_simple_basic_avro kafka_simple_handle_key_only kafka_simple_handle_key_only_avro kafka_simple_claim_check kafka_simple_claim_check_avro canal_json_adapter_compatibility canal_json_basic canal_json_content_compatible multi_topics avro_basic canal_json_handle_key_only open_protocol_handle_key_only canal_json_claim_check open_protocol_claim_check canal_json_storage_basic canal_json_storage_partition_table multi_tables_ddl PROW_JOB_ID=cd0f73e1-2b01-4d7c-b838-cad499e0f373 JENKINS_NODE_COOKIE=1fcb848a-18c1-48e6-b501-96585363270d BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-t8cg2 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test","buildid":"1787445299229233158","prowjobid":"cd0f73e1-2b01-4d7c-b838-cad499e0f373","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/usr/java/openjdk-17/bin:/usr/local/pulsar/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#72 TEST_GROUP=G01 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233158 GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-07t56 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-49db5c2ac89dce8b54933296f8c1a6663a0148762a9e3dd665403ece56205519 NODE_LABELS=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-07t56 ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-t8cg2 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-t8cg2 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=72 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/http_api/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:45:00 CST 2024] <<<<<< run test case http_api success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/http_api_tls/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:45:03 CST 2024] <<<<<< run test case http_api_tls success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/api_v2/run.sh using Sink-Type: pulsar... <<================= find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/http_api_tls_with_user_auth/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:45:10 CST 2024] <<<<<< run test case http_api_tls_with_user_auth success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/cli_tls_with_auth/run.sh using Sink-Type: pulsar... <<================= find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_simple_basic/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:45:16 CST 2024] <<<<<< run test case kafka_simple_basic success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_simple_basic_avro/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:45:19 CST 2024] <<<<<< run test case kafka_simple_basic_avro success! >>>>>> Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc) 3667975680 bytes in 18.03 secs (203458444 bytes/sec) [Pipeline] { [Pipeline] sh [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] node [Pipeline] node + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G02 Run cases: consistent_replicate_ddl consistent_replicate_gbk consistent_replicate_nfs consistent_replicate_storage_file consistent_replicate_storage_file_large_value consistent_replicate_storage_s3 consistent_partition_table kafka_big_messages_v2 multi_tables_ddl_v2 multi_topics_v2 storage_cleanup csv_storage_basic csv_storage_multi_tables_ddl csv_storage_partition_table PROW_JOB_ID=cd0f73e1-2b01-4d7c-b838-cad499e0f373 JENKINS_NODE_COOKIE=4903d9a4-85b0-4d11-b274-050571beb691 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-z1579 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test","buildid":"1787445299229233158","prowjobid":"cd0f73e1-2b01-4d7c-b838-cad499e0f373","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/usr/java/openjdk-17/bin:/usr/local/pulsar/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#72 TEST_GROUP=G02 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233158 GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-m6xj7 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-49db5c2ac89dce8b54933296f8c1a6663a0148762a9e3dd665403ece56205519 NODE_LABELS=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-m6xj7 ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-z1579 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-z1579 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=72 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/consistent_replicate_ddl/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:45:21 CST 2024] <<<<<< run test case consistent_replicate_ddl success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] checkout [Pipeline] { The recommended git tool is: git [Pipeline] checkout [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] stage [Pipeline] { (Test) The recommended git tool is: git [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-txbrj [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-qx8wr [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-96v7q [Pipeline] { Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-08gz3 [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-807c6 [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-z5vmx [Pipeline] { [Pipeline] cache No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@5b2e3c2d; decorates RemoteLauncher[hudson.remoting.Channel@70533516:JNLP4-connect connection from 10.233.108.30/10.233.108.30:56728] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@704b5ece; decorates RemoteLauncher[hudson.remoting.Channel@23298966:JNLP4-connect connection from 10.233.93.57/10.233.93.57:59730] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@396632be; decorates RemoteLauncher[hudson.remoting.Channel@4bb90545:JNLP4-connect connection from 10.233.97.251/10.233.97.251:59414] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@1bed3edc; decorates RemoteLauncher[hudson.remoting.Channel@2e958732:JNLP4-connect connection from 10.233.127.100/10.233.127.100:55552] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_simple_handle_key_only/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:45:23 CST 2024] <<<<<< run test case kafka_simple_handle_key_only success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/consistent_replicate_gbk/run.sh using Sink-Type: pulsar... <<================= * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:24927; Connection refused * Closing connection 0 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Commit message: "fix(br): use failpoint tidb-server instead (#2951)" Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Agent ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-txbrj is provisioned from template ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-5flxm-srqcj --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "33eb7c5bcd041ebbfad07972382de6247aa4b23f" jenkins/label: "ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-5flxm" name: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-txbrj" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:pulsar-test" name: "golang" resources: limits: memory: "16Gi" cpu: "8" requests: memory: "16Gi" cpu: "8" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-txbrj" - name: "JENKINS_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-txbrj" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-txbrj in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test Commit message: "fix(br): use failpoint tidb-server instead (#2951)" Commit message: "fix(br): use failpoint tidb-server instead (#2951)" You are running an older version of MinIO released 3 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide Agent ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-96v7q is provisioned from template ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-n4fbv-v43fl --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "99ab3771e4f88b961b49b6a3c654c684d0aa86ba" jenkins/label: "ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-n4fbv" name: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-96v7q" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:pulsar-test" name: "golang" resources: limits: memory: "16Gi" cpu: "8" requests: memory: "16Gi" cpu: "8" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-96v7q" - name: "JENKINS_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-96v7q" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 Running on ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-96v7q in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:24927 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 17CCE3B581340736 < X-Xss-Protection: 1; mode=block < Date: Mon, 06 May 2024 11:45:26 GMT < { [data not shown] * Connection #0 to host 127.0.0.1 left intact find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_simple_handle_key_only_avro/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:45:26 CST 2024] <<<<<< run test case kafka_simple_handle_key_only_avro success! >>>>>> Commit message: "fix(br): use failpoint tidb-server instead (#2951)" Bucket 's3://logbucket/' created [Mon May 6 19:45:26 CST 2024] <<<<<< run test case consistent_replicate_gbk success! >>>>>> Exiting on signal: INTERRUPT Agent ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-08gz3 is provisioned from template ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-r63qz-2hx65 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "3bb51e2983ce0d308360ec97ac9ea8e0ff8cf323" jenkins/label: "ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-r63qz" name: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-08gz3" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:pulsar-test" name: "golang" resources: limits: memory: "16Gi" cpu: "8" requests: memory: "16Gi" cpu: "8" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-08gz3" - name: "JENKINS_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-08gz3" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-08gz3 in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test Agent ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-qx8wr is provisioned from template ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-wpwbl-pl6f9 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "8778ec358850d9aec0f3ae915ac6f9a6cc9e544b" jenkins/label: "ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-wpwbl" name: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-qx8wr" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:pulsar-test" name: "golang" resources: limits: memory: "16Gi" cpu: "8" requests: memory: "16Gi" cpu: "8" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-qx8wr" - name: "JENKINS_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-qx8wr" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-qx8wr in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_simple_claim_check/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:45:29 CST 2024] <<<<<< run test case kafka_simple_claim_check success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/consistent_replicate_nfs/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:45:30 CST 2024] <<<<<< run test case consistent_replicate_nfs success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Agent ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-807c6 is provisioned from template ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-ddztn-jqhc2 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "2575736ee1a8ff4350bc8f4984879e828a4ade89" jenkins/label: "ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-ddztn" name: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-807c6" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:pulsar-test" name: "golang" resources: limits: memory: "16Gi" cpu: "8" requests: memory: "16Gi" cpu: "8" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-807c6" - name: "JENKINS_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-807c6" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-807c6 in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test Agent ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-z5vmx is provisioned from template ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-cb666-j880z --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" runUrl: "job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "c1ea813e09b0f68a10d6a5b0b057defb65e35bae" jenkins/label: "ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-cb666" name: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-z5vmx" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:pulsar-test" name: "golang" resources: limits: memory: "16Gi" cpu: "8" requests: memory: "16Gi" cpu: "8" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-z5vmx" - name: "JENKINS_NAME" value: "ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-z5vmx" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_simple_claim_check_avro/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:45:32 CST 2024] <<<<<< run test case kafka_simple_claim_check_avro success! >>>>>> Running on ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-z5vmx in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/consistent_replicate_storage_file/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:45:33 CST 2024] <<<<<< run test case consistent_replicate_storage_file success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/canal_json_adapter_compatibility/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:45:35 CST 2024] <<<<<< run test case canal_json_adapter_compatibility success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/consistent_replicate_storage_file_large_value/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:45:36 CST 2024] <<<<<< run test case consistent_replicate_storage_file_large_value success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/canal_json_basic/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/consistent_replicate_storage_s3/run.sh using Sink-Type: pulsar... <<================= * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:24927; Connection refused * Closing connection 0 You are running an older version of MinIO released 3 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:24927 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 17CCE3B8E4E948F8 < X-Xss-Protection: 1; mode=block < Date: Mon, 06 May 2024 11:45:41 GMT < { [data not shown] * Connection #0 to host 127.0.0.1 left intact Bucket 's3://logbucket/' created [Mon May 6 19:45:41 CST 2024] <<<<<< run test case consistent_replicate_storage_s3 success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Exiting on signal: INTERRUPT start tidb cluster in /tmp/tidb_cdc_test/canal_json_basic Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/consistent_partition_table/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:45:44 CST 2024] <<<<<< run test case consistent_partition_table success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc) 3667975680 bytes in 20.56 secs (178431009 bytes/sec) [Pipeline] { [Pipeline] cache Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_big_messages_v2/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:45:47 CST 2024] <<<<<< run test case kafka_big_messages_v2 success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/multi_tables_ddl_v2/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:45:50 CST 2024] <<<<<< run test case multi_tables_ddl_v2 success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/multi_topics_v2/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:45:53 CST 2024] <<<<<< run test case multi_topics_v2 success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36e5f5bc0003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-t8cg2, pid:2229, start at 2024-05-06 19:45:53.008621717 +0800 CST m=+5.108747926 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:47:53.016 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:45:53.007 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:35:53.007 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36e5f5bc0003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-t8cg2, pid:2229, start at 2024-05-06 19:45:53.008621717 +0800 CST m=+5.108747926 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:47:53.016 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:45:53.007 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:35:53.007 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36e5f7300014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-t8cg2, pid:2302, start at 2024-05-06 19:45:53.149127202 +0800 CST m=+5.198136092 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:47:53.156 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:45:53.150 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:35:53.150 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/canal_json_basic/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/canal_json_basic/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/canal_json_basic/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/canal_json_basic/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/canal_json_basic/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Mon May 6 19:45:56 CST 2024] <<<<<< START cdc server in canal_json_basic case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_basic.36863688.out server --log-file /tmp/tidb_cdc_test/canal_json_basic/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/canal_json_basic/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc) 3667975680 bytes in 7.74 secs (473808382 bytes/sec) [Pipeline] { [Pipeline] cache find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/storage_cleanup/run.sh using Sink-Type: pulsar... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/storage_cleanup/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/storage_cleanup ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/storage_cleanup + source /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/storage_cleanup/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/storage_cleanup + CDC_BINARY=cdc.test + SINK_TYPE=pulsar + EXIST_FILES=() + CLEANED_FILES=() + trap stop_tidb_cluster EXIT + run pulsar + '[' pulsar '!=' storage ']' + return + check_logs /tmp/tidb_cdc_test/storage_cleanup ++ date + echo '[Mon May 6 19:45:57 CST 2024] <<<<<< run test case storage_cleanup success! >>>>>>' [Mon May 6 19:45:57 CST 2024] <<<<<< run test case storage_cleanup success! >>>>>> + stop_tidb_cluster + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:45:59 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1370990d-97c7-4157-830c-f0124cbe5fd0 {"id":"1370990d-97c7-4157-830c-f0124cbe5fd0","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714995956} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4db9539c28 1370990d-97c7-4157-830c-f0124cbe5fd0 /tidb/cdc/default/default/upstream/7365851492814834982 {"id":7365851492814834982,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1370990d-97c7-4157-830c-f0124cbe5fd0 {"id":"1370990d-97c7-4157-830c-f0124cbe5fd0","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714995956} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4db9539c28 1370990d-97c7-4157-830c-f0124cbe5fd0 /tidb/cdc/default/default/upstream/7365851492814834982 {"id":7365851492814834982,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1370990d-97c7-4157-830c-f0124cbe5fd0 {"id":"1370990d-97c7-4157-830c-f0124cbe5fd0","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714995956} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4db9539c28 1370990d-97c7-4157-830c-f0124cbe5fd0 /tidb/cdc/default/default/upstream/7365851492814834982 {"id":7365851492814834982,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:45:59 CST 2024] <<<<<< START pulsar cluster in canal_json_basic case >>>>>>' [Mon May 6 19:45:59 CST 2024] <<<<<< START pulsar cluster in canal_json_basic case >>>>>> + workdir=/tmp/tidb_cdc_test/canal_json_basic + cluster_type=normal + cd /tmp/tidb_cdc_test/canal_json_basic + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/canal_json_basic/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/canal_json_basic/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/canal_json_basic/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/canal_json_basic/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/canal_json_basic/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/canal_json_basic/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/canal_json_basic/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/canal_json_basic/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/canal_json_basic/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/canal_json_basic/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/canal_json_basic/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/canal_json_basic/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:45:59 CST 2024] <<<<<< START pulsar cluster in normal mode in canal_json_basic case >>>>>>' [Mon May 6 19:45:59 CST 2024] <<<<<< START pulsar cluster in normal mode in canal_json_basic case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/canal_json_basic/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/canal_json_basic/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/canal_json_basic/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/csv_storage_basic/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:46:00 CST 2024] <<<<<< run test case csv_storage_basic success! >>>>>> + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:46:03 CST 2024] <<<<<< run test case csv_storage_multi_tables_ddl success! >>>>>> Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc) 3667975680 bytes in 5.45 secs (672738311 bytes/sec) [Pipeline] { [Pipeline] cache public/default ++ date + echo '[Mon May 6 19:46:05 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:46:05 CST 2024] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_basic.cli.4135.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-canal-json-basic?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: 80e5bc7c-e3a3-41bc-bfae-2e902160017d Info: {"upstream_id":7365851492814834982,"namespace":"default","id":"80e5bc7c-e3a3-41bc-bfae-2e902160017d","sink_uri":"pulsar://127.0.0.1:6650/ticdc-canal-json-basic?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:46:06.634524588+08:00","start_ts":449575902526046212,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575902526046212,"checkpoint_ts":449575902526046212,"checkpoint_time":"2024-05-06 19:46:06.057"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/csv_storage_partition_table/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:46:06 CST 2024] <<<<<< run test case csv_storage_partition_table success! >>>>>> + set +x \033[0;36m<<< Run all test success >>>\033[0m [Mon May 6 19:46:13 CST 2024] <<<<<< START Pulsar consumer in canal_json_basic case >>>>>> Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc) 3667975680 bytes in 7.20 secs (509183930 bytes/sec) [Pipeline] { [Pipeline] cache Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc) 3667975680 bytes in 5.84 secs (628213239 bytes/sec) [Pipeline] { [Pipeline] cache table test.finish_mark not exists for 1-th check, retry later table test.finish_mark not exists for 2-th check, retry later table test.finish_mark not exists for 3-th check, retry later table test.finish_mark not exists for 4-th check, retry later table test.finish_mark not exists for 5-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc) 3667975680 bytes in 14.04 secs (261330108 bytes/sec) [Pipeline] { [Pipeline] sh table test.finish_mark not exists for 6-th check, retry later [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G03 Run cases: row_format drop_many_tables processor_stop_delay partition_table PROW_JOB_ID=cd0f73e1-2b01-4d7c-b838-cad499e0f373 JENKINS_NODE_COOKIE=5ebb706f-1ec5-43de-996d-56502cbb268c BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-k03d6 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test","buildid":"1787445299229233158","prowjobid":"cd0f73e1-2b01-4d7c-b838-cad499e0f373","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/usr/java/openjdk-17/bin:/usr/local/pulsar/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#72 TEST_GROUP=G03 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233158 GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-sd01c GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-49db5c2ac89dce8b54933296f8c1a6663a0148762a9e3dd665403ece56205519 NODE_LABELS=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-sd01c ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-k03d6 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-k03d6 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=72 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/row_format/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G04 Run cases: foreign_key ddl_puller_lag ddl_only_block_related_table changefeed_auto_stop PROW_JOB_ID=cd0f73e1-2b01-4d7c-b838-cad499e0f373 JENKINS_NODE_COOKIE=4dba4fb8-2159-462d-9b22-03b5d4f1639a BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-55h44 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test","buildid":"1787445299229233158","prowjobid":"cd0f73e1-2b01-4d7c-b838-cad499e0f373","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/usr/java/openjdk-17/bin:/usr/local/pulsar/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#72 TEST_GROUP=G04 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233158 GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-5hwc4 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-49db5c2ac89dce8b54933296f8c1a6663a0148762a9e3dd665403ece56205519 NODE_LABELS=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-5hwc4 ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-55h44 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-55h44 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=72 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/foreign_key/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G00 Run cases: bdr_mode capture_suicide_while_balance_table syncpoint hang_sink_suicide server_config_compatibility kafka_big_messages kafka_compression kafka_messages kafka_sink_error_resume mq_sink_lost_callback mq_sink_dispatcher kafka_column_selector kafka_column_selector_avro debezium lossy_ddl storage_csv_update PROW_JOB_ID=cd0f73e1-2b01-4d7c-b838-cad499e0f373 JENKINS_NODE_COOKIE=de5f0761-b726-4801-9b2c-f8370817dd79 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-105q1 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test","buildid":"1787445299229233158","prowjobid":"cd0f73e1-2b01-4d7c-b838-cad499e0f373","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/usr/java/openjdk-17/bin:/usr/local/pulsar/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#72 TEST_GROUP=G00 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233158 GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-pccmc GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-49db5c2ac89dce8b54933296f8c1a6663a0148762a9e3dd665403ece56205519 NODE_LABELS=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-105q1 ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-pccmc GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-105q1 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=72 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/bdr_mode/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:46:40 CST 2024] <<<<<< run test case bdr_mode success! >>>>>> [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G06 Run cases: sink_retry changefeed_error ddl_sequence resourcecontrol PROW_JOB_ID=cd0f73e1-2b01-4d7c-b838-cad499e0f373 JENKINS_NODE_COOKIE=6e285ae5-ae80-4b0e-8f9b-1d4f2cd4865b BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jzh76 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test","buildid":"1787445299229233158","prowjobid":"cd0f73e1-2b01-4d7c-b838-cad499e0f373","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/usr/java/openjdk-17/bin:/usr/local/pulsar/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#72 TEST_GROUP=G06 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233158 GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-ppcz6 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-49db5c2ac89dce8b54933296f8c1a6663a0148762a9e3dd665403ece56205519 NODE_LABELS=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-ppcz6 ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jzh76 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jzh76 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=72 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/sink_retry/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table test.finish_mark not exists for 7-th check, retry later [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G07 Run cases: kv_client_stream_reconnect cdc split_region PROW_JOB_ID=cd0f73e1-2b01-4d7c-b838-cad499e0f373 JENKINS_NODE_COOKIE=6a46b50a-fe0f-4ed6-8bd6-70e207a7c503 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-fqtgg HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test","buildid":"1787445299229233158","prowjobid":"cd0f73e1-2b01-4d7c-b838-cad499e0f373","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/usr/java/openjdk-17/bin:/usr/local/pulsar/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#72 TEST_GROUP=G07 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233158 GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-lj75x GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-49db5c2ac89dce8b54933296f8c1a6663a0148762a9e3dd665403ece56205519 NODE_LABELS=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-lj75x ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-fqtgg GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-fqtgg GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=72 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kv_client_stream_reconnect/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G05 [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc already exists) Run cases: charset_gbk ddl_manager multi_source PROW_JOB_ID=cd0f73e1-2b01-4d7c-b838-cad499e0f373 JENKINS_NODE_COOKIE=fe62e7a9-7435-4757-be1a-434cee7939ec BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-b9q8w HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test","buildid":"1787445299229233158","prowjobid":"cd0f73e1-2b01-4d7c-b838-cad499e0f373","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/usr/java/openjdk-17/bin:/usr/local/pulsar/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#72 TEST_GROUP=G05 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233158 GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-gdv1f GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-49db5c2ac89dce8b54933296f8c1a6663a0148762a9e3dd665403ece56205519 NODE_LABELS=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-b9q8w ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-gdv1f GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-b9q8w GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=72 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/charset_gbk/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/row_format Starting Upstream PD... [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { table test.finish_mark not exists for 8-th check, retry later Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/foreign_key Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... [Pipeline] { [Pipeline] { find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/capture_suicide_while_balance_table/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:46:44 CST 2024] <<<<<< run test case capture_suicide_while_balance_table success! >>>>>> [Pipeline] checkout The recommended git tool is: git [Pipeline] // cache [Pipeline] } [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git The recommended git tool is: git [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] // dir [Pipeline] } [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] // timeout [Pipeline] } [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] cache start tidb cluster in /tmp/tidb_cdc_test/sink_retry Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@2f0451a7; decorates RemoteLauncher[hudson.remoting.Channel@2ab07741:JNLP4-connect connection from 10.233.72.36/10.233.72.36:55910] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@43e0a568; decorates RemoteLauncher[hudson.remoting.Channel@3c1cb4dc:JNLP4-connect connection from 10.233.84.19/10.233.84.19:54320] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@733c98bd; decorates RemoteLauncher[hudson.remoting.Channel@57df2a43:JNLP4-connect connection from 10.233.67.11/10.233.67.11:51544] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 table test.finish_mark not exists for 9-th check, retry later Cloning repository https://github.com/PingCAP-QE/ci.git No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@3dba7482; decorates RemoteLauncher[hudson.remoting.Channel@10067412:JNLP4-connect connection from 10.233.105.209/10.233.105.209:48206] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@2496f30c; decorates RemoteLauncher[hudson.remoting.Channel@6a22a89a:JNLP4-connect connection from 10.233.90.159/10.233.90.159:37482] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@79466151; decorates RemoteLauncher[hudson.remoting.Channel@729b11c:JNLP4-connect connection from 10.233.68.33/10.233.68.33:45060] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/syncpoint/run.sh using Sink-Type: pulsar... <<================= kafka downstream isn't support syncpoint record [Mon May 6 19:46:47 CST 2024] <<<<<< run test case syncpoint success! >>>>>> start tidb cluster in /tmp/tidb_cdc_test/charset_gbk Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/kv_client_stream_reconnect Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 table test.finish_mark not exists for 10-th check, retry later Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Avoid second fetch Commit message: "fix(br): use failpoint tidb-server instead (#2951)" Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Commit message: "fix(br): use failpoint tidb-server instead (#2951)" Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 Commit message: "fix(br): use failpoint tidb-server instead (#2951)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Commit message: "fix(br): use failpoint tidb-server instead (#2951)" Verifying downstream PD is started... Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Commit message: "fix(br): use failpoint tidb-server instead (#2951)" table test.finish_mark not exists for 11-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/hang_sink_suicide/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:46:50 CST 2024] <<<<<< run test case hang_sink_suicide success! >>>>>> table test.finish_mark exists check diff successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36e9a22c000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-k03d6, pid:1335, start at 2024-05-06 19:46:53.207589654 +0800 CST m=+5.373027373 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:48:53.214 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:46:53.195 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:36:53.195 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/server_config_compatibility/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:46:54 CST 2024] <<<<<< run test case server_config_compatibility success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36e9b3b80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-55h44, pid:1333, start at 2024-05-06 19:46:54.341154538 +0800 CST m=+5.228992926 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:48:54.349 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:46:54.318 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:36:54.318 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36e9a22c000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-k03d6, pid:1335, start at 2024-05-06 19:46:53.207589654 +0800 CST m=+5.373027373 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:48:53.214 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:46:53.195 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:36:53.195 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36e9a2f80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-k03d6, pid:1417, start at 2024-05-06 19:46:53.276816787 +0800 CST m=+5.389642028 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:48:53.283 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:46:53.246 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:36:53.246 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/row_format/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/row_format/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/row_format/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/row_format/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/row_format/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36e9c5c00017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jzh76, pid:1221, start at 2024-05-06 19:46:55.507528212 +0800 CST m=+5.245208689 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:48:55.514 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:46:55.472 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:36:55.472 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36e9c5c00017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jzh76, pid:1221, start at 2024-05-06 19:46:55.507528212 +0800 CST m=+5.245208689 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:48:55.514 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:46:55.472 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:36:55.472 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36e9c8e40003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jzh76, pid:1307, start at 2024-05-06 19:46:55.675883123 +0800 CST m=+5.355894148 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:48:55.682 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:46:55.673 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:36:55.673 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sink_retry/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sink_retry/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Avoid second fetch Checking out Revision 03312178c534dce949face80c69812d989e55009 (origin/main) Commit message: "fix(br): use failpoint tidb-server instead (#2951)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 03312178c534dce949face80c69812d989e55009 # timeout=10 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_big_messages/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:46:57 CST 2024] <<<<<< run test case kafka_big_messages success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36e9b3b80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-55h44, pid:1333, start at 2024-05-06 19:46:54.341154538 +0800 CST m=+5.228992926 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:48:54.349 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:46:54.318 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:36:54.318 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36e9b47c0015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-55h44, pid:1415, start at 2024-05-06 19:46:54.40365555 +0800 CST m=+5.236169703 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:48:54.411 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:46:54.416 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:36:54.416 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/foreign_key/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/foreign_key/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36e9d7f80002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-fqtgg, pid:1281, start at 2024-05-06 19:46:56.639038509 +0800 CST m=+5.154976954 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:48:56.646 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:46:56.638 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:36:56.638 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36e9d7f80002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-fqtgg, pid:1281, start at 2024-05-06 19:46:56.639038509 +0800 CST m=+5.154976954 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:48:56.646 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:46:56.638 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:36:56.638 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36e9db080008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-fqtgg, pid:1359, start at 2024-05-06 19:46:56.84151661 +0800 CST m=+5.303031863 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:48:56.850 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:46:56.834 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:36:56.834 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.cli.2695.out cli tso query --pd=http://127.0.0.1:2379 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36e9e90c0016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-b9q8w, pid:1283, start at 2024-05-06 19:46:57.763839233 +0800 CST m=+5.055996613 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:48:57.769 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:46:57.731 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:36:57.731 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36e9e90c0016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-b9q8w, pid:1283, start at 2024-05-06 19:46:57.763839233 +0800 CST m=+5.055996613 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:48:57.769 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:46:57.731 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:36:57.731 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36e9ec200014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-b9q8w, pid:1371, start at 2024-05-06 19:46:57.97650559 +0800 CST m=+5.206083437 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:48:57.983 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:46:57.978 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:36:57.978 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/charset_gbk/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/charset_gbk/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.cli.2691.out cli tso query --pd=http://127.0.0.1:2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.cli.2687.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449575916272353281 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575916272353281 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:46:59 CST 2024] <<<<<< START cdc server in row_format case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.27302732.out server --log-file /tmp/tidb_cdc_test/row_format/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/row_format/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ++ date + echo '[Mon May 6 19:47:00 CST 2024] <<<<<< START pulsar cluster in kv_client_stream_reconnect case >>>>>>' [Mon May 6 19:47:00 CST 2024] <<<<<< START pulsar cluster in kv_client_stream_reconnect case >>>>>> + workdir=/tmp/tidb_cdc_test/kv_client_stream_reconnect + cluster_type=oauth + cd /tmp/tidb_cdc_test/kv_client_stream_reconnect + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/kv_client_stream_reconnect/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/kv_client_stream_reconnect/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/kv_client_stream_reconnect/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/kv_client_stream_reconnect/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/kv_client_stream_reconnect/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/kv_client_stream_reconnect/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/kv_client_stream_reconnect/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/kv_client_stream_reconnect/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/kv_client_stream_reconnect/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/kv_client_stream_reconnect/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/kv_client_stream_reconnect/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/kv_client_stream_reconnect/pulsar_standalone.conf + pulsar_port=6650 + '[' oauth == mtls ']' + '[' oauth == oauth ']' + echo 'Waiting for oauth2 server to be ready...' Waiting for oauth2 server to be ready... + i=0 + nc -z localhost 9096 + oauth2-server + gen_oauth_config + echo ' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' + echo ' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/kv_client_stream_reconnect/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/kv_client_stream_reconnect/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ date + echo '[Mon May 6 19:47:00 CST 2024] <<<<<< START pulsar cluster in oauth mode in kv_client_stream_reconnect case >>>>>>' [Mon May 6 19:47:00 CST 2024] <<<<<< START pulsar cluster in oauth mode in kv_client_stream_reconnect case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/kv_client_stream_reconnect/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/kv_client_stream_reconnect/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/kv_client_stream_reconnect/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 + set +x + tso='449575916397395969 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575916397395969 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ***************** properties ***************** "readproportion"="0" "mysql.user"="root" "updateproportion"="0" "threadcount"="2" "scanproportion"="0" "workload"="core" "mysql.host"="127.0.0.1" "mysql.port"="4000" "readallfields"="true" "requestdistribution"="uniform" "recordcount"="10" "mysql.db"="sink_retry" "operationcount"="0" "insertproportion"="0" "dotransactions"="false" ********************************************** Run finished, takes 9.12478ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2110.6, Avg(us): 1729, Min(us): 959, Max(us): 4514, 95th(us): 5000, 99th(us): 5000 [Mon May 6 19:47:00 CST 2024] <<<<<< START cdc server in sink_retry case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkTxnRandomError=25%return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.27552757.out server --log-file /tmp/tidb_cdc_test/sink_retry/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sink_retry/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_compression/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:47:00 CST 2024] <<<<<< run test case kafka_compression success! >>>>>> + set +x + tso='449575916592693249 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575916592693249 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:47:01 CST 2024] <<<<<< START cdc server in foreign_key case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.27212723.out server --log-file /tmp/tidb_cdc_test/foreign_key/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/foreign_key/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Mon May 6 19:47:01 CST 2024] <<<<<< START cdc server in charset_gbk case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.charset_gbk.27572759.out server --log-file /tmp/tidb_cdc_test/charset_gbk/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/charset_gbk/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc) 3667975680 bytes in 15.76 secs (232704775 bytes/sec) [Pipeline] { [Pipeline] cache + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:47:03 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d47f715b-aaae-4062-bd67-f4674cb1f0ff {"id":"d47f715b-aaae-4062-bd67-f4674cb1f0ff","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996020} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dba439b3a d47f715b-aaae-4062-bd67-f4674cb1f0ff /tidb/cdc/default/default/upstream/7365851751412397007 {"id":7365851751412397007,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d47f715b-aaae-4062-bd67-f4674cb1f0ff {"id":"d47f715b-aaae-4062-bd67-f4674cb1f0ff","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996020} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dba439b3a d47f715b-aaae-4062-bd67-f4674cb1f0ff /tidb/cdc/default/default/upstream/7365851751412397007 {"id":7365851751412397007,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d47f715b-aaae-4062-bd67-f4674cb1f0ff {"id":"d47f715b-aaae-4062-bd67-f4674cb1f0ff","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996020} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dba439b3a d47f715b-aaae-4062-bd67-f4674cb1f0ff /tidb/cdc/default/default/upstream/7365851751412397007 {"id":7365851751412397007,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:47:03 CST 2024] <<<<<< START pulsar cluster in row_format case >>>>>>' [Mon May 6 19:47:03 CST 2024] <<<<<< START pulsar cluster in row_format case >>>>>> + workdir=/tmp/tidb_cdc_test/row_format + cluster_type=normal + cd /tmp/tidb_cdc_test/row_format + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/row_format/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/row_format/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/row_format/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/row_format/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/row_format/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/row_format/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/row_format/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/row_format/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/row_format/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/row_format/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/row_format/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/row_format/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:47:03 CST 2024] <<<<<< START pulsar cluster in normal mode in row_format case >>>>>>' [Mon May 6 19:47:03 CST 2024] <<<<<< START pulsar cluster in normal mode in row_format case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/row_format/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/row_format/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/row_format/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:47:03 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/eac37833-14d8-4498-9e79-ca58c33ae247 {"id":"eac37833-14d8-4498-9e79-ca58c33ae247","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996020} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dba473847 eac37833-14d8-4498-9e79-ca58c33ae247 /tidb/cdc/default/default/upstream/7365851763190632460 {"id":7365851763190632460,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/eac37833-14d8-4498-9e79-ca58c33ae247 {"id":"eac37833-14d8-4498-9e79-ca58c33ae247","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996020} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dba473847 eac37833-14d8-4498-9e79-ca58c33ae247 /tidb/cdc/default/default/upstream/7365851763190632460 {"id":7365851763190632460,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/eac37833-14d8-4498-9e79-ca58c33ae247 {"id":"eac37833-14d8-4498-9e79-ca58c33ae247","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996020} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dba473847 eac37833-14d8-4498-9e79-ca58c33ae247 /tidb/cdc/default/default/upstream/7365851763190632460 {"id":7365851763190632460,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:47:03 CST 2024] <<<<<< START pulsar cluster in sink_retry case >>>>>>' [Mon May 6 19:47:03 CST 2024] <<<<<< START pulsar cluster in sink_retry case >>>>>> + workdir=/tmp/tidb_cdc_test/sink_retry + cluster_type=normal + cd /tmp/tidb_cdc_test/sink_retry + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/sink_retry/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/sink_retry/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/sink_retry/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/sink_retry/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/sink_retry/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/sink_retry/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/sink_retry/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/sink_retry/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/sink_retry/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/sink_retry/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/sink_retry/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/sink_retry/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:47:03 CST 2024] <<<<<< START pulsar cluster in normal mode in sink_retry case >>>>>>' [Mon May 6 19:47:03 CST 2024] <<<<<< START pulsar cluster in normal mode in sink_retry case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/sink_retry/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/sink_retry/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/sink_retry/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_messages/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:47:03 CST 2024] <<<<<< run test case kafka_messages success! >>>>>> table test.finish_mark not exists for 1-th check, retry later + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:47:04 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/189eedfa-7f11-4c7d-83e7-593b2cd75277 {"id":"189eedfa-7f11-4c7d-83e7-593b2cd75277","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996021} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dba44ad3b 189eedfa-7f11-4c7d-83e7-593b2cd75277 /tidb/cdc/default/default/upstream/7365851758807902179 {"id":7365851758807902179,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/189eedfa-7f11-4c7d-83e7-593b2cd75277 {"id":"189eedfa-7f11-4c7d-83e7-593b2cd75277","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996021} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dba44ad3b 189eedfa-7f11-4c7d-83e7-593b2cd75277 /tidb/cdc/default/default/upstream/7365851758807902179 {"id":7365851758807902179,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/189eedfa-7f11-4c7d-83e7-593b2cd75277 {"id":"189eedfa-7f11-4c7d-83e7-593b2cd75277","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996021} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dba44ad3b 189eedfa-7f11-4c7d-83e7-593b2cd75277 /tidb/cdc/default/default/upstream/7365851758807902179 {"id":7365851758807902179,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:47:04 CST 2024] <<<<<< START pulsar cluster in foreign_key case >>>>>>' [Mon May 6 19:47:04 CST 2024] <<<<<< START pulsar cluster in foreign_key case >>>>>> + workdir=/tmp/tidb_cdc_test/foreign_key + cluster_type=normal + cd /tmp/tidb_cdc_test/foreign_key + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/foreign_key/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/foreign_key/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/foreign_key/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/foreign_key/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/foreign_key/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/foreign_key/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/foreign_key/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/foreign_key/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/foreign_key/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/foreign_key/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/foreign_key/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/foreign_key/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:47:04 CST 2024] <<<<<< START pulsar cluster in normal mode in foreign_key case >>>>>>' [Mon May 6 19:47:04 CST 2024] <<<<<< START pulsar cluster in normal mode in foreign_key case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/foreign_key/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/foreign_key/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/foreign_key/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:47:04 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/821df3a5-ea73-433e-a9c1-d5a608187755 {"id":"821df3a5-ea73-433e-a9c1-d5a608187755","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996021} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dba50a230 821df3a5-ea73-433e-a9c1-d5a608187755 /tidb/cdc/default/default/upstream/7365851770852988169 {"id":7365851770852988169,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/821df3a5-ea73-433e-a9c1-d5a608187755 {"id":"821df3a5-ea73-433e-a9c1-d5a608187755","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996021} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dba50a230 821df3a5-ea73-433e-a9c1-d5a608187755 /tidb/cdc/default/default/upstream/7365851770852988169 {"id":7365851770852988169,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/821df3a5-ea73-433e-a9c1-d5a608187755 {"id":"821df3a5-ea73-433e-a9c1-d5a608187755","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996021} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dba50a230 821df3a5-ea73-433e-a9c1-d5a608187755 /tidb/cdc/default/default/upstream/7365851770852988169 {"id":7365851770852988169,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:47:04 CST 2024] <<<<<< START pulsar cluster in charset_gbk case >>>>>>' [Mon May 6 19:47:04 CST 2024] <<<<<< START pulsar cluster in charset_gbk case >>>>>> + workdir=/tmp/tidb_cdc_test/charset_gbk + cluster_type=mtls + cd /tmp/tidb_cdc_test/charset_gbk + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/charset_gbk/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/charset_gbk/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/charset_gbk/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/charset_gbk/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/charset_gbk/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/charset_gbk/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/charset_gbk/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/charset_gbk/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/charset_gbk/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/charset_gbk/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/charset_gbk/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/charset_gbk/pulsar_standalone.conf + pulsar_port=6650 + '[' mtls == mtls ']' + pulsar_port=6651 + gen_mtls_config + openssl genrsa -out ca.key.pem 2048 Generating RSA private key, 2048 bit long modulus .......................................................................+++ ......................................+++ e is 65537 (0x10001) + openssl req -x509 -new -nodes -key ca.key.pem -subj /CN=CARoot -days 365 -out ca.cert.pem + openssl genrsa -out server.key.pem 2048 Generating RSA private key, 2048 bit long modulus ..+++ .............+++ e is 65537 (0x10001) + openssl pkcs8 -topk8 -inform PEM -outform PEM -in server.key.pem -out server.key-pk8.pem -nocrypt + echo '[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + openssl req -new -config server.conf -key server.key.pem -out server.csr.pem -sha256 + openssl x509 -req -in server.csr.pem -CA ca.cert.pem -CAkey ca.key.pem -CAcreateserial -out server.cert.pem -days 365 -extensions v3_ext -extfile server.conf -sha256 Signature ok subject=/CN=server Getting CA Private Key + openssl genrsa -out broker_client.key.pem 2048 Generating RSA private key, 2048 bit long modulus .....................+++ ...........................................................................................+++ e is 65537 (0x10001) + openssl pkcs8 -topk8 -inform PEM -outform PEM -in broker_client.key.pem -out broker_client.key-pk8.pem -nocrypt + openssl req -new -subj /CN=broker_client -key broker_client.key.pem -out broker_client.csr.pem -sha256 + openssl x509 -req -in broker_client.csr.pem -CA ca.cert.pem -CAkey ca.key.pem -CAcreateserial -out broker_client.cert.pem -days 365 -sha256 Signature ok subject=/CN=broker_client Getting CA Private Key + echo ' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/charset_gbk/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/charset_gbk/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/charset_gbk/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/charset_gbk/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/charset_gbk/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/charset_gbk/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' + echo ' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/charset_gbk/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/charset_gbk/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/charset_gbk/ca.cert.pem' ++ date + echo '[Mon May 6 19:47:04 CST 2024] <<<<<< START pulsar cluster in mtls mode in charset_gbk case >>>>>>' [Mon May 6 19:47:04 CST 2024] <<<<<< START pulsar cluster in mtls mode in charset_gbk case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6651 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/charset_gbk/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/charset_gbk/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/charset_gbk/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 table test.finish_mark not exists for 2-th check, retry later + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_sink_error_resume/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:47:06 CST 2024] <<<<<< run test case kafka_sink_error_resume success! >>>>>> + nc -z localhost 6651 + i=2 + '[' 2 -gt 20 ']' + sleep 2 public/default table test.finish_mark not exists for 3-th check, retry later ++ date + echo '[Mon May 6 19:47:06 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:47:06 CST 2024] <<<<<< pulsar is ready >>>>>> [Mon May 6 19:47:06 CST 2024] <<<<<< START cdc server in kv_client_stream_reconnect case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/kv/kvClientForceReconnect=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kv_client_stream_reconnect.30903092.out server --log-file /tmp/tidb_cdc_test/kv_client_stream_reconnect/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kv_client_stream_reconnect/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + nc -z localhost 6650 + i=3 + '[' 3 -gt 20 ']' + sleep 2 + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + nc -z localhost 6651 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public table test.finish_mark not exists for 4-th check, retry later public/default ++ date + echo '[Mon May 6 19:47:09 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:47:09 CST 2024] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.cli.3174.out cli changefeed create --start-ts=449575916272353281 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-row-format-test-28583?protocol=canal-json&enable-tidb-extension=true' find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/mq_sink_lost_callback/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:47:09 CST 2024] <<<<<< run test case mq_sink_lost_callback success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:47:10 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/738bce28-a2fb-4ee2-981f-d4cf1f335ca3 {"id":"738bce28-a2fb-4ee2-981f-d4cf1f335ca3","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996027} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dba51d54a 738bce28-a2fb-4ee2-981f-d4cf1f335ca3 /tidb/cdc/default/default/upstream/7365851768516924775 {"id":7365851768516924775,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/738bce28-a2fb-4ee2-981f-d4cf1f335ca3 {"id":"738bce28-a2fb-4ee2-981f-d4cf1f335ca3","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996027} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dba51d54a 738bce28-a2fb-4ee2-981f-d4cf1f335ca3 /tidb/cdc/default/default/upstream/7365851768516924775 {"id":7365851768516924775,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/738bce28-a2fb-4ee2-981f-d4cf1f335ca3 {"id":"738bce28-a2fb-4ee2-981f-d4cf1f335ca3","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996027} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dba51d54a 738bce28-a2fb-4ee2-981f-d4cf1f335ca3 /tidb/cdc/default/default/upstream/7365851768516924775 {"id":7365851768516924775,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + nc -z localhost 6650 + i=4 + '[' 4 -gt 20 ']' + sleep 2 Create changefeed successfully! ID: 88f6b140-1c9d-4f6b-a366-7186fb841de8 Info: {"upstream_id":7365851751412397007,"namespace":"default","id":"88f6b140-1c9d-4f6b-a366-7186fb841de8","sink_uri":"pulsar://127.0.0.1:6650/ticdc-row-format-test-28583?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:47:10.556307288+08:00","start_ts":449575916272353281,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575916272353281,"checkpoint_ts":449575916272353281,"checkpoint_time":"2024-05-06 19:46:58.495"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... [Mon May 6 19:47:10 CST 2024] <<<<<< START Pulsar consumer in kv_client_stream_reconnect case >>>>>> table test.finish_mark exists public/default ++ date + echo '[Mon May 6 19:47:10 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:47:10 CST 2024] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.cli.3176.out cli changefeed create --start-ts=449575916592693249 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-foreign-key-test-5414?protocol=canal-json&enable-tidb-extension=true' check diff successfully wait process cdc.test exit for 1-th time... Create changefeed successfully! ID: 7c8b18c0-0925-4f09-a7b2-ef55a2a361f0 Info: {"upstream_id":7365851758807902179,"namespace":"default","id":"7c8b18c0-0925-4f09-a7b2-ef55a2a361f0","sink_uri":"pulsar://127.0.0.1:6650/ticdc-foreign-key-test-5414?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:47:11.90935776+08:00","start_ts":449575916592693249,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575916592693249,"checkpoint_ts":449575916592693249,"checkpoint_time":"2024-05-06 19:46:59.717"} PASS + set +x [Mon May 6 19:47:12 CST 2024] <<<<<< START Pulsar consumer in row_format case >>>>>> coverage: 2.6% of statements in github.com/pingcap/tiflow/... + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:47:13 CST 2024] <<<<<< run test case canal_json_basic success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/mq_sink_dispatcher/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:47:12 CST 2024] <<<<<< run test case mq_sink_dispatcher success! >>>>>> + set +x [Mon May 6 19:47:13 CST 2024] <<<<<< START Pulsar consumer in foreign_key case >>>>>> public/default ++ date + echo '[Mon May 6 19:47:14 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:47:14 CST 2024] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.cli.3217.out cli changefeed create --start-ts=449575916397395969 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-sink-retry-test-19030?protocol=canal-json&enable-tidb-extension=true' public/default ++ date + echo '[Mon May 6 19:47:14 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:47:14 CST 2024] <<<<<< pulsar is ready >>>>>> Create changefeed successfully! ID: 7d99a06a-de7c-4016-95db-e7972fed7440 Info: {"upstream_id":7365851770852988169,"namespace":"default","id":"7d99a06a-de7c-4016-95db-e7972fed7440","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-05-06T19:47:14.489863481+08:00","start_ts":449575916963364865,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575916963364865,"checkpoint_ts":449575916963364865,"checkpoint_time":"2024-05-06 19:47:01.131"} [Mon May 6 19:47:14 CST 2024] <<<<<< START Pulsar consumer in charset_gbk case >>>>>> table foreign_key.finish_mark not exists for 1-th check, retry later Create changefeed successfully! ID: 79e60770-3a5b-49df-8a8f-a77fff6734ca Info: {"upstream_id":7365851763190632460,"namespace":"default","id":"79e60770-3a5b-49df-8a8f-a77fff6734ca","sink_uri":"pulsar://127.0.0.1:6650/ticdc-sink-retry-test-19030?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:47:15.507203001+08:00","start_ts":449575916397395969,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575916397395969,"checkpoint_ts":449575916397395969,"checkpoint_time":"2024-05-06 19:46:58.972"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_column_selector/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:47:15 CST 2024] <<<<<< run test case kafka_column_selector success! >>>>>> + set +x [Mon May 6 19:47:16 CST 2024] <<<<<< START Pulsar consumer in sink_retry case >>>>>> table foreign_key.finish_mark not exists for 2-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/kafka_column_selector_avro/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:47:19 CST 2024] <<<<<< run test case kafka_column_selector_avro success! >>>>>> table foreign_key.finish_mark not exists for 3-th check, retry later table foreign_key.finish_mark not exists for 4-th check, retry later table row_format.finish_mark not exists for 1-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/debezium/run.sh using Sink-Type: pulsar... <<================= find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/lossy_ddl/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:47:22 CST 2024] <<<<<< run test case lossy_ddl success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/canal_json_content_compatible/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table charset_gbk_test0.t0 exists table charset_gbk_test0.t1 exists table charset_gbk_test1.t0 not exists for 1-th check, retry later table row_format.finish_mark not exists for 2-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc) 3667975680 bytes in 17.77 secs (206446929 bytes/sec) table foreign_key.finish_mark not exists for 5-th check, retry later [Pipeline] { [Pipeline] cache table charset_gbk_test1.t0 exists table test.finish_mark not exists for 1-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/storage_csv_update/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:47:25 CST 2024] <<<<<< run test case storage_csv_update success! >>>>>> table foreign_key.finish_mark not exists for 6-th check, retry later table row_format.finish_mark not exists for 3-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/canal_json_content_compatible Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... table test.finish_mark not exists for 2-th check, retry later table row_format.finish_mark not exists for 4-th check, retry later table foreign_key.finish_mark not exists for 7-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release \033[0;36m<<< Run all test success >>>\033[0m table test.finish_mark not exists for 3-th check, retry later table row_format.finish_mark not exists for 5-th check, retry later table foreign_key.finish_mark not exists for 8-th check, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 4-th check, retry later table row_format.finish_mark not exists for 6-th check, retry later table foreign_key.finish_mark not exists for 9-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark exists check table exists success check diff successfully Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc) 3667975680 bytes in 7.27 secs (504846763 bytes/sec) [Pipeline] { [Pipeline] cache wait process cdc.test exit for 1-th time... table row_format.finish_mark not exists for 7-th check, retry later table foreign_key.finish_mark not exists for 10-th check, retry later wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:47:34 CST 2024] <<<<<< run test case charset_gbk success! >>>>>> table foreign_key.finish_mark not exists for 11-th check, retry later table row_format.finish_mark not exists for 8-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ec370c000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-t8cg2, pid:5501, start at 2024-05-06 19:47:35.510062575 +0800 CST m=+5.178911328 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:49:35.518 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:47:35.491 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:37:35.491 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table foreign_key.finish_mark exists check diff successfully table row_format.finish_mark not exists for 9-th check, retry later wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ec370c000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-t8cg2, pid:5501, start at 2024-05-06 19:47:35.510062575 +0800 CST m=+5.178911328 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:49:35.518 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:47:35.491 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:37:35.491 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ec37b40003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-t8cg2, pid:5585, start at 2024-05-06 19:47:35.53537567 +0800 CST m=+5.149662857 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:49:35.541 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:47:35.533 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:37:35.533 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/canal_json_content_compatible/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/canal_json_content_compatible/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/canal_json_content_compatible/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/canal_json_content_compatible/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/canal_json_content_compatible/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:47:39 CST 2024] <<<<<< run test case foreign_key success! >>>>>> table row_format.finish_mark not exists for 10-th check, retry later [Mon May 6 19:47:40 CST 2024] <<<<<< START cdc server in canal_json_content_compatible case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_content_compatible.68746876.out server --log-file /tmp/tidb_cdc_test/canal_json_content_compatible/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/canal_json_content_compatible/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table row_format.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:47:43 CST 2024] <<<<<< run test case row_format success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:47:43 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c8d2f752-4ed1-4dfe-a34a-aefa52188c09 {"id":"c8d2f752-4ed1-4dfe-a34a-aefa52188c09","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996061} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbae99b32 c8d2f752-4ed1-4dfe-a34a-aefa52188c09 /tidb/cdc/default/default/upstream/7365851935920427530 {"id":7365851935920427530,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c8d2f752-4ed1-4dfe-a34a-aefa52188c09 {"id":"c8d2f752-4ed1-4dfe-a34a-aefa52188c09","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996061} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbae99b32 c8d2f752-4ed1-4dfe-a34a-aefa52188c09 /tidb/cdc/default/default/upstream/7365851935920427530 {"id":7365851935920427530,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c8d2f752-4ed1-4dfe-a34a-aefa52188c09 {"id":"c8d2f752-4ed1-4dfe-a34a-aefa52188c09","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996061} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbae99b32 c8d2f752-4ed1-4dfe-a34a-aefa52188c09 /tidb/cdc/default/default/upstream/7365851935920427530 {"id":7365851935920427530,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:47:43 CST 2024] <<<<<< START pulsar cluster in canal_json_content_compatible case >>>>>>' [Mon May 6 19:47:43 CST 2024] <<<<<< START pulsar cluster in canal_json_content_compatible case >>>>>> + workdir=/tmp/tidb_cdc_test/canal_json_content_compatible + cluster_type=normal + cd /tmp/tidb_cdc_test/canal_json_content_compatible + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/canal_json_content_compatible/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/canal_json_content_compatible/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/canal_json_content_compatible/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/canal_json_content_compatible/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/canal_json_content_compatible/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/canal_json_content_compatible/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/canal_json_content_compatible/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/canal_json_content_compatible/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/canal_json_content_compatible/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/canal_json_content_compatible/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/canal_json_content_compatible/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/canal_json_content_compatible/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:47:43 CST 2024] <<<<<< START pulsar cluster in normal mode in canal_json_content_compatible case >>>>>>' [Mon May 6 19:47:43 CST 2024] <<<<<< START pulsar cluster in normal mode in canal_json_content_compatible case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/canal_json_content_compatible/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/canal_json_content_compatible/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/canal_json_content_compatible/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/ddl_manager/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public table sink_retry.finish_mark_1 exists check diff successfully ***************** properties ***************** "mysql.host"="127.0.0.1" "recordcount"="10" "workload"="core" "mysql.db"="sink_retry" "readproportion"="0" "operationcount"="0" "mysql.port"="4000" "updateproportion"="0" "dotransactions"="false" "threadcount"="2" "mysql.user"="root" "requestdistribution"="uniform" "readallfields"="true" "insertproportion"="0" "scanproportion"="0" ********************************************** Run finished, takes 400.094791ms INSERT - Takes(s): 0.4, Count: 10, OPS: 25.1, Avg(us): 40399, Min(us): 592, Max(us): 395014, 95th(us): 396000, 99th(us): 396000 check diff failed 1-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/ddl_puller_lag/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... public/default start tidb cluster in /tmp/tidb_cdc_test/ddl_manager Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... ++ date + echo '[Mon May 6 19:47:50 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:47:50 CST 2024] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_content_compatible.cli.7326.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-canal-json-content-compatible?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: d1decb98-32a6-468d-aaa1-2a3cb1dfa218 Info: {"upstream_id":7365851935920427530,"namespace":"default","id":"d1decb98-32a6-468d-aaa1-2a3cb1dfa218","sink_uri":"pulsar://127.0.0.1:6650/ticdc-canal-json-content-compatible?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:47:51.044369932+08:00","start_ts":449575929902792710,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575929902792710,"checkpoint_ts":449575929902792710,"checkpoint_time":"2024-05-06 19:47:50.491"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... check diff successfully Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc) 3667975680 bytes in 15.72 secs (233365416 bytes/sec) [Pipeline] { [Pipeline] sh wait process cdc.test exit for 1-th time... [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G08 Run cases: processor_err_chan changefeed_reconstruct multi_capture synced_status_with_redo PROW_JOB_ID=cd0f73e1-2b01-4d7c-b838-cad499e0f373 JENKINS_NODE_COOKIE=abbb5cea-5a7c-421a-a71c-370623a01ecc BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test","buildid":"1787445299229233158","prowjobid":"cd0f73e1-2b01-4d7c-b838-cad499e0f373","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/usr/java/openjdk-17/bin:/usr/local/pulsar/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#72 TEST_GROUP=G08 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233158 GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-spg9v GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-49db5c2ac89dce8b54933296f8c1a6663a0148762a9e3dd665403ece56205519 NODE_LABELS=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6 ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-spg9v GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=72 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/processor_err_chan/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh wait process cdc.test exit for 2-th time... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G09 Run cases: gc_safepoint changefeed_pause_resume cli_with_auth savepoint synced_status PROW_JOB_ID=cd0f73e1-2b01-4d7c-b838-cad499e0f373 JENKINS_NODE_COOKIE=c65d40c7-f9ee-4163-aa8f-1a01e8f348c6 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test","buildid":"1787445299229233158","prowjobid":"cd0f73e1-2b01-4d7c-b838-cad499e0f373","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/usr/java/openjdk-17/bin:/usr/local/pulsar/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#72 TEST_GROUP=G09 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233158 GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-nxj9w GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-49db5c2ac89dce8b54933296f8c1a6663a0148762a9e3dd665403ece56205519 NODE_LABELS=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-nxj9w GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=72 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/gc_safepoint/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G10 Run cases: default_value simple cdc_server_tips event_filter sql_mode PROW_JOB_ID=cd0f73e1-2b01-4d7c-b838-cad499e0f373 JENKINS_NODE_COOKIE=2d5d7947-4f7b-49e2-80c2-61f23ad395ce BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test","buildid":"1787445299229233158","prowjobid":"cd0f73e1-2b01-4d7c-b838-cad499e0f373","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/usr/java/openjdk-17/bin:/usr/local/pulsar/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#72 TEST_GROUP=G10 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233158 GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-7n0m1 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-49db5c2ac89dce8b54933296f8c1a6663a0148762a9e3dd665403ece56205519 NODE_LABELS=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-7n0m1 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=72 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/default_value/run.sh using Sink-Type: pulsar... <<================= + set +x wait process cdc.test exit for 3-th time... [Pipeline] // stage [Pipeline] } + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G11 Run cases: resolve_lock move_table autorandom generate_column PROW_JOB_ID=cd0f73e1-2b01-4d7c-b838-cad499e0f373 JENKINS_NODE_COOKIE=dec493ae-e054-494e-8b93-8b9d973e61a1 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-v1jxz HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test","buildid":"1787445299229233158","prowjobid":"cd0f73e1-2b01-4d7c-b838-cad499e0f373","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/usr/java/openjdk-17/bin:/usr/local/pulsar/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#72 TEST_GROUP=G11 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233158 GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-4nhnt GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-49db5c2ac89dce8b54933296f8c1a6663a0148762a9e3dd665403ece56205519 NODE_LABELS=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-v1jxz ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-4nhnt GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-v1jxz GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=72 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/resolve_lock/run.sh using Sink-Type: pulsar... <<================= start tidb cluster in /tmp/tidb_cdc_test/ddl_puller_lag Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:47:53 CST 2024] <<<<<< run test case kv_client_stream_reconnect success! >>>>>> Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/drop_many_tables/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/gc_safepoint Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/processor_err_chan Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... The 1 times to try to start tidb cluster... find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/simple/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/drop_many_tables Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc already exists) Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Mon May 6 19:47:57 CST 2024] <<<<<< START Pulsar consumer in canal_json_content_compatible case >>>>>> Verifying downstream PD is started... [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] // container [Pipeline] } [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // cache [Pipeline] } Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // withEnv [Pipeline] } [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] // dir [Pipeline] } [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { start tidb cluster in /tmp/tidb_cdc_test/resolve_lock Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [Pipeline] // node [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow [Pipeline] { start tidb cluster in /tmp/tidb_cdc_test/simple Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] cache Verifying downstream PD is started... Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36edba240006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-b9q8w, pid:4328, start at 2024-05-06 19:48:00.271573868 +0800 CST m=+5.060654875 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:00.278 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:00.265 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:00.265 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36edba240006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-b9q8w, pid:4328, start at 2024-05-06 19:48:00.271573868 +0800 CST m=+5.060654875 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:00.278 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:00.265 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:00.265 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36edbc64000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-b9q8w, pid:4404, start at 2024-05-06 19:48:00.419349116 +0800 CST m=+5.158305609 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:00.426 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:00.409 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:00.409 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_manager/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_manager/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36edd8f80017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-55h44, pid:4381, start at 2024-05-06 19:48:02.284441466 +0800 CST m=+5.156845954 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:02.291 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:02.289 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:02.289 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36edd8f80017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-55h44, pid:4381, start at 2024-05-06 19:48:02.284441466 +0800 CST m=+5.156845954 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:02.291 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:02.289 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:02.289 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36edd9980014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-55h44, pid:4462, start at 2024-05-06 19:48:02.324668189 +0800 CST m=+5.146803215 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:02.331 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:02.327 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:02.327 +0800 All versions after safe point can be accessed. (DO NOT EDIT) table test.finish_mark not exists for 1-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Mon May 6 19:48:03 CST 2024] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.58065808.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 2-th check, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.cli.5808.out cli tso query --pd=http://127.0.0.1:2379 Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:48:06 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/920fcf74-02cc-4897-9350-0daaf7d905a8 {"id":"920fcf74-02cc-4897-9350-0daaf7d905a8","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996083} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb46de29 920fcf74-02cc-4897-9350-0daaf7d905a8 /tidb/cdc/default/default/upstream/7365852043233991414 {"id":7365852043233991414,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/920fcf74-02cc-4897-9350-0daaf7d905a8 {"id":"920fcf74-02cc-4897-9350-0daaf7d905a8","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996083} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb46de29 920fcf74-02cc-4897-9350-0daaf7d905a8 /tidb/cdc/default/default/upstream/7365852043233991414 {"id":7365852043233991414,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/920fcf74-02cc-4897-9350-0daaf7d905a8 {"id":"920fcf74-02cc-4897-9350-0daaf7d905a8","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996083} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb46de29 920fcf74-02cc-4897-9350-0daaf7d905a8 /tidb/cdc/default/default/upstream/7365852043233991414 {"id":7365852043233991414,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:48:06 CST 2024] <<<<<< START pulsar cluster in ddl_manager case >>>>>>' [Mon May 6 19:48:06 CST 2024] <<<<<< START pulsar cluster in ddl_manager case >>>>>> + workdir=/tmp/tidb_cdc_test/ddl_manager + cluster_type=normal + cd /tmp/tidb_cdc_test/ddl_manager + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_manager/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/ddl_manager/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/ddl_manager/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_manager/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/ddl_manager/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/ddl_manager/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/ddl_manager/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/ddl_manager/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_manager/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/ddl_manager/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/ddl_manager/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/ddl_manager/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:48:06 CST 2024] <<<<<< START pulsar cluster in normal mode in ddl_manager case >>>>>>' [Mon May 6 19:48:06 CST 2024] <<<<<< START pulsar cluster in normal mode in ddl_manager case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/ddl_manager/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/ddl_manager/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/ddl_manager/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ee13280013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb, pid:1332, start at 2024-05-06 19:48:05.983521735 +0800 CST m=+5.216258062 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:05.991 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:05.962 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:05.962 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + tso='449575933899702273 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575933899702273 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ee18e00013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6, pid:1331, start at 2024-05-06 19:48:06.364305356 +0800 CST m=+5.206904843 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:06.371 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:06.328 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:06.328 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 3-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/cdc/run.sh using Sink-Type: pulsar... <<================= ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Mon May 6 19:48:08 CST 2024] <<<<<< START cdc server in ddl_puller_lag case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorDDLResolved=1*sleep(180000)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.58585860.out server --log-file /tmp/tidb_cdc_test/ddl_puller_lag/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_puller_lag/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ee22900016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-k03d6, pid:4419, start at 2024-05-06 19:48:06.968192957 +0800 CST m=+5.181750634 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:06.976 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:06.948 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:06.948 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ee13280013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb, pid:1332, start at 2024-05-06 19:48:05.983521735 +0800 CST m=+5.216258062 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:05.991 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:05.962 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:05.962 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ee13380018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb, pid:1426, start at 2024-05-06 19:48:06.015761531 +0800 CST m=+5.190299267 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:06.022 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:06.016 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:06.016 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/gc_safepoint/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/gc_safepoint/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ee18e00013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6, pid:1331, start at 2024-05-06 19:48:06.364305356 +0800 CST m=+5.206904843 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:06.371 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:06.328 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:06.328 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ee18d00018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6, pid:1416, start at 2024-05-06 19:48:06.373870576 +0800 CST m=+5.146791056 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:06.380 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:06.373 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:06.373 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_err_chan/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_err_chan/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table test.finish_mark not exists for 4-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ee22900016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-k03d6, pid:4419, start at 2024-05-06 19:48:06.968192957 +0800 CST m=+5.181750634 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:06.976 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:06.948 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:06.948 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ee24180013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-k03d6, pid:4498, start at 2024-05-06 19:48:07.069754706 +0800 CST m=+5.222935326 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:07.076 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:07.046 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:07.046 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/drop_many_tables/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/drop_many_tables/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/split_region/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table test.finish_mark not exists for 5-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:48:11 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/afc53d09-60a4-465c-9ade-a0693f0c3c45 {"id":"afc53d09-60a4-465c-9ade-a0693f0c3c45","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996088} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb505451 afc53d09-60a4-465c-9ade-a0693f0c3c45 /tidb/cdc/default/default/upstream/7365852054844484137 {"id":7365852054844484137,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/afc53d09-60a4-465c-9ade-a0693f0c3c45 {"id":"afc53d09-60a4-465c-9ade-a0693f0c3c45","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996088} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb505451 afc53d09-60a4-465c-9ade-a0693f0c3c45 /tidb/cdc/default/default/upstream/7365852054844484137 {"id":7365852054844484137,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/afc53d09-60a4-465c-9ade-a0693f0c3c45 {"id":"afc53d09-60a4-465c-9ade-a0693f0c3c45","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996088} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb505451 afc53d09-60a4-465c-9ade-a0693f0c3c45 /tidb/cdc/default/default/upstream/7365852054844484137 {"id":7365852054844484137,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:48:11 CST 2024] <<<<<< START pulsar cluster in ddl_puller_lag case >>>>>>' [Mon May 6 19:48:11 CST 2024] <<<<<< START pulsar cluster in ddl_puller_lag case >>>>>> + workdir=/tmp/tidb_cdc_test/ddl_puller_lag + cluster_type=normal + cd /tmp/tidb_cdc_test/ddl_puller_lag + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_puller_lag/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/ddl_puller_lag/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/ddl_puller_lag/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_puller_lag/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/ddl_puller_lag/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/ddl_puller_lag/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/ddl_puller_lag/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/ddl_puller_lag/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_puller_lag/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/ddl_puller_lag/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/ddl_puller_lag/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/ddl_puller_lag/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:48:11 CST 2024] <<<<<< START pulsar cluster in normal mode in ddl_puller_lag case >>>>>>' [Mon May 6 19:48:11 CST 2024] <<<<<< START pulsar cluster in normal mode in ddl_puller_lag case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/ddl_puller_lag/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/ddl_puller_lag/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/ddl_puller_lag/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ee534c0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh, pid:1426, start at 2024-05-06 19:48:10.090887121 +0800 CST m=+5.085049431 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:10.097 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:10.067 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:10.067 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ee534c0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh, pid:1426, start at 2024-05-06 19:48:10.090887121 +0800 CST m=+5.085049431 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:10.097 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:10.067 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:10.067 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ee55a80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh, pid:1514, start at 2024-05-06 19:48:10.237910889 +0800 CST m=+5.174840324 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:10.244 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:10.218 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:10.218 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/simple/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/simple/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/simple/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/simple/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/simple/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ++ date + echo '[Mon May 6 19:48:11 CST 2024] <<<<<< START pulsar cluster in gc_safepoint case >>>>>>' [Mon May 6 19:48:11 CST 2024] <<<<<< START pulsar cluster in gc_safepoint case >>>>>> + workdir=/tmp/tidb_cdc_test/gc_safepoint + cluster_type=normal + cd /tmp/tidb_cdc_test/gc_safepoint + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/gc_safepoint/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/gc_safepoint/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/gc_safepoint/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/gc_safepoint/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/gc_safepoint/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/gc_safepoint/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/gc_safepoint/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/gc_safepoint/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/gc_safepoint/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/gc_safepoint/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/gc_safepoint/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/gc_safepoint/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:48:11 CST 2024] <<<<<< START pulsar cluster in normal mode in gc_safepoint case >>>>>>' [Mon May 6 19:48:11 CST 2024] <<<<<< START pulsar cluster in normal mode in gc_safepoint case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/gc_safepoint/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/gc_safepoint/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/gc_safepoint/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ee60100005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-v1jxz, pid:1426, start at 2024-05-06 19:48:10.889161233 +0800 CST m=+5.125369890 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:10.894 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:10.884 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:10.884 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ee60100005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-v1jxz, pid:1426, start at 2024-05-06 19:48:10.889161233 +0800 CST m=+5.125369890 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:10.894 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:10.884 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:10.884 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ee61980014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-v1jxz, pid:1508, start at 2024-05-06 19:48:11.013745426 +0800 CST m=+5.195812969 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:11.020 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:10.982 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:10.982 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ++ date + echo '[Mon May 6 19:48:11 CST 2024] <<<<<< START pulsar cluster in processor_err_chan case >>>>>>' [Mon May 6 19:48:11 CST 2024] <<<<<< START pulsar cluster in processor_err_chan case >>>>>> + workdir=/tmp/tidb_cdc_test/processor_err_chan + cluster_type=normal + cd /tmp/tidb_cdc_test/processor_err_chan + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/processor_err_chan/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/processor_err_chan/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/processor_err_chan/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/processor_err_chan/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/processor_err_chan/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/processor_err_chan/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/processor_err_chan/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/processor_err_chan/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/processor_err_chan/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/processor_err_chan/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/processor_err_chan/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/processor_err_chan/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:48:11 CST 2024] <<<<<< START pulsar cluster in normal mode in processor_err_chan case >>>>>>' [Mon May 6 19:48:11 CST 2024] <<<<<< START pulsar cluster in normal mode in processor_err_chan case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/processor_err_chan/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/processor_err_chan/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/processor_err_chan/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/resolve_lock/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/resolve_lock/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.cli.5834.out cli tso query --pd=http://127.0.0.1:2379 Tenant not found Reason: Tenant not found + i=1 + '[' 1 -gt 20 ']' + sleep 2 table test.finish_mark not exists for 6-th check, retry later + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.cli.2793.out cli tso query --pd=http://127.0.0.1:2379 + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 + set +x + tso='449575935658950658 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575935658950658 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:48:13 CST 2024] <<<<<< START cdc server in drop_many_tables case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.58755877.out server --log-file /tmp/tidb_cdc_test/drop_many_tables/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/drop_many_tables/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 start tidb cluster in /tmp/tidb_cdc_test/split_region Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.cli.2797.out cli tso query --pd=http://127.0.0.1:2379 + /usr/local/pulsar/bin/pulsar-admin namespaces list public table test.finish_mark not exists for 7-th check, retry later + set +x + tso='449575935965134849 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575935965134849 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:48:15 CST 2024] <<<<<< START cdc server in simple case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.28442846.out server --log-file /tmp/tidb_cdc_test/simple/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/simple/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + set +x + tso='449575936179306497 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575936179306497 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:48:15 CST 2024] <<<<<< START cdc server in resolve_lock case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.28372839.out server --log-file /tmp/tidb_cdc_test/resolve_lock/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/resolve_lock/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc) 3667975680 bytes in 12.46 secs (294264226 bytes/sec) [Pipeline] { [Pipeline] cache + nc -z localhost 6650 + i=3 + '[' 3 -gt 20 ']' + sleep 2 table test.finish_mark not exists for 8-th check, retry later Tenant not found Reason: Tenant not found + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:48:17 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/59b05518-96b3-4bf7-99ba-3882366656e6 {"id":"59b05518-96b3-4bf7-99ba-3882366656e6","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996094} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb60933c 59b05518-96b3-4bf7-99ba-3882366656e6 /tidb/cdc/default/default/upstream/7365852072008343416 {"id":7365852072008343416,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/59b05518-96b3-4bf7-99ba-3882366656e6 {"id":"59b05518-96b3-4bf7-99ba-3882366656e6","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996094} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb60933c 59b05518-96b3-4bf7-99ba-3882366656e6 /tidb/cdc/default/default/upstream/7365852072008343416 {"id":7365852072008343416,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/59b05518-96b3-4bf7-99ba-3882366656e6 {"id":"59b05518-96b3-4bf7-99ba-3882366656e6","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996094} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb60933c 59b05518-96b3-4bf7-99ba-3882366656e6 /tidb/cdc/default/default/upstream/7365852072008343416 {"id":7365852072008343416,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:48:17 CST 2024] <<<<<< START pulsar cluster in drop_many_tables case >>>>>>' [Mon May 6 19:48:17 CST 2024] <<<<<< START pulsar cluster in drop_many_tables case >>>>>> + workdir=/tmp/tidb_cdc_test/drop_many_tables + cluster_type=normal + cd /tmp/tidb_cdc_test/drop_many_tables + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/drop_many_tables/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/drop_many_tables/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/drop_many_tables/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/drop_many_tables/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/drop_many_tables/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/drop_many_tables/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/drop_many_tables/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/drop_many_tables/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/drop_many_tables/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/drop_many_tables/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/drop_many_tables/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/drop_many_tables/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:48:17 CST 2024] <<<<<< START pulsar cluster in normal mode in drop_many_tables case >>>>>>' [Mon May 6 19:48:17 CST 2024] <<<<<< START pulsar cluster in normal mode in drop_many_tables case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/drop_many_tables/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/drop_many_tables/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/drop_many_tables/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 + i=1 + '[' 1 -gt 20 ']' + sleep 2 public/default ++ date + echo '[Mon May 6 19:48:17 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:48:17 CST 2024] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.cli.6337.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-12528?protocol=canal-json&enable-tidb-extension=true' -c=ddl-manager public/default ++ date + echo '[Mon May 6 19:48:17 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:48:17 CST 2024] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.cli.6300.out cli changefeed create --start-ts=449575933899702273 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-ddl-puller-lag-test-31123?protocol=canal-json&enable-tidb-extension=true' Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:48:18 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3aa86c6e-7b9e-4ce2-9d61-6a3a2b3191fb {"id":"3aa86c6e-7b9e-4ce2-9d61-6a3a2b3191fb","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996095} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb6f1b47 3aa86c6e-7b9e-4ce2-9d61-6a3a2b3191fb /tidb/cdc/default/default/upstream/7365852082963684363 {"id":7365852082963684363,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3aa86c6e-7b9e-4ce2-9d61-6a3a2b3191fb {"id":"3aa86c6e-7b9e-4ce2-9d61-6a3a2b3191fb","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996095} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb6f1b47 3aa86c6e-7b9e-4ce2-9d61-6a3a2b3191fb /tidb/cdc/default/default/upstream/7365852082963684363 {"id":7365852082963684363,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3aa86c6e-7b9e-4ce2-9d61-6a3a2b3191fb {"id":"3aa86c6e-7b9e-4ce2-9d61-6a3a2b3191fb","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996095} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb6f1b47 3aa86c6e-7b9e-4ce2-9d61-6a3a2b3191fb /tidb/cdc/default/default/upstream/7365852082963684363 {"id":7365852082963684363,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:48:18 CST 2024] <<<<<< START pulsar cluster in simple case >>>>>>' [Mon May 6 19:48:18 CST 2024] <<<<<< START pulsar cluster in simple case >>>>>> + workdir=/tmp/tidb_cdc_test/simple + cluster_type=normal + cd /tmp/tidb_cdc_test/simple + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/simple/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/simple/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/simple/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/simple/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/simple/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/simple/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/simple/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/simple/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/simple/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/simple/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/simple/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/simple/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:48:18 CST 2024] <<<<<< START pulsar cluster in normal mode in simple case >>>>>>' [Mon May 6 19:48:18 CST 2024] <<<<<< START pulsar cluster in normal mode in simple case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/simple/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/simple/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/simple/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 Create changefeed successfully! ID: a0b39556-a91b-4973-9786-15e3955a01ba Info: {"upstream_id":7365852054844484137,"namespace":"default","id":"a0b39556-a91b-4973-9786-15e3955a01ba","sink_uri":"pulsar://127.0.0.1:6650/ticdc-ddl-puller-lag-test-31123?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:48:18.803827395+08:00","start_ts":449575933899702273,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575933899702273,"checkpoint_ts":449575933899702273,"checkpoint_time":"2024-05-06 19:48:05.738"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:48:18 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/699ff630-49de-42ce-a167-2f832ce0d445 {"id":"699ff630-49de-42ce-a167-2f832ce0d445","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996096} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb6c0e3a 699ff630-49de-42ce-a167-2f832ce0d445 /tidb/cdc/default/default/upstream/7365852080212050206 {"id":7365852080212050206,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/699ff630-49de-42ce-a167-2f832ce0d445 {"id":"699ff630-49de-42ce-a167-2f832ce0d445","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996096} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb6c0e3a 699ff630-49de-42ce-a167-2f832ce0d445 /tidb/cdc/default/default/upstream/7365852080212050206 {"id":7365852080212050206,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/699ff630-49de-42ce-a167-2f832ce0d445 {"id":"699ff630-49de-42ce-a167-2f832ce0d445","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996096} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb6c0e3a 699ff630-49de-42ce-a167-2f832ce0d445 /tidb/cdc/default/default/upstream/7365852080212050206 {"id":7365852080212050206,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:48:18 CST 2024] <<<<<< START pulsar cluster in resolve_lock case >>>>>>' [Mon May 6 19:48:18 CST 2024] <<<<<< START pulsar cluster in resolve_lock case >>>>>> + workdir=/tmp/tidb_cdc_test/resolve_lock + cluster_type=normal + cd /tmp/tidb_cdc_test/resolve_lock + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/resolve_lock/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/resolve_lock/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/resolve_lock/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/resolve_lock/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/resolve_lock/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/resolve_lock/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/resolve_lock/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/resolve_lock/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/resolve_lock/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/resolve_lock/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/resolve_lock/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/resolve_lock/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:48:19 CST 2024] <<<<<< START pulsar cluster in normal mode in resolve_lock case >>>>>>' [Mon May 6 19:48:19 CST 2024] <<<<<< START pulsar cluster in normal mode in resolve_lock case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/resolve_lock/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/resolve_lock/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/resolve_lock/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 + /usr/local/pulsar/bin/pulsar-admin namespaces list public table test.finish_mark not exists for 9-th check, retry later public/default + set +x [Mon May 6 19:48:20 CST 2024] <<<<<< START Pulsar consumer in ddl_puller_lag case >>>>>> Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ++ date + echo '[Mon May 6 19:48:20 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:48:20 CST 2024] <<<<<< pulsar is ready >>>>>> + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 table sink_retry.finish_mark_2 exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:48:21 CST 2024] <<<<<< run test case sink_retry success! >>>>>> public/default ++ date + echo '[Mon May 6 19:48:21 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:48:21 CST 2024] <<<<<< pulsar is ready >>>>>> [Mon May 6 19:48:21 CST 2024] <<<<<< START cdc server in gc_safepoint case >>>>>> table test.finish_mark not exists for 10-th check, retry later Create changefeed successfully! ID: ddl-manager Info: {"upstream_id":7365852043233991414,"namespace":"default","id":"ddl-manager","sink_uri":"pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-12528?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:48:21.572726011+08:00","start_ts":449575937039663106,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575937039663106,"checkpoint_ts":449575937039663106,"checkpoint_time":"2024-05-06 19:48:17.716"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/txnutil/gc/InjectGcSafepointUpdateInterval=return(500)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.gc_safepoint.31783180.out server --log-file /tmp/tidb_cdc_test/gc_safepoint/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/gc_safepoint/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + set +x [Mon May 6 19:48:23 CST 2024] <<<<<< START Pulsar consumer in ddl_manager case >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 11-th check, retry later public/default ++ date + echo '[Mon May 6 19:48:23 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:48:23 CST 2024] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.cli.6330.out cli changefeed create --start-ts=449575935658950658 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-drop-tables-test-10625?protocol=canal-json&enable-tidb-extension=true' [Mon May 6 19:48:23 CST 2024] <<<<<< START cdc server in processor_err_chan case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/ProcessorAddTableError=1*return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_err_chan.32393241.out server --log-file /tmp/tidb_cdc_test/processor_err_chan/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_err_chan/cdc_data --cluster-id default --config /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/processor_err_chan/conf/server.toml --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Create changefeed successfully! ID: 003b0f06-8fcd-42f2-a4c4-5c19f80887af Info: {"upstream_id":7365852072008343416,"namespace":"default","id":"003b0f06-8fcd-42f2-a4c4-5c19f80887af","sink_uri":"pulsar://127.0.0.1:6650/ticdc-drop-tables-test-10625?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:48:24.575183607+08:00","start_ts":449575935658950658,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575935658950658,"checkpoint_ts":449575935658950658,"checkpoint_time":"2024-05-06 19:48:12.449"} PASS public/default coverage: 2.6% of statements in github.com/pingcap/tiflow/... ++ date + echo '[Mon May 6 19:48:24 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:48:24 CST 2024] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.cli.3291.out cli changefeed create --start-ts=449575935965134849 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-simple-test-28323?protocol=canal-json&enable-tidb-extension=true' + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:48:24 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/42debffc-1ae6-42b4-ae40-7e52eabdc87b {"id":"42debffc-1ae6-42b4-ae40-7e52eabdc87b","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996101} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb5c8862 42debffc-1ae6-42b4-ae40-7e52eabdc87b /tidb/cdc/default/default/upstream/7365852066098678294 {"id":7365852066098678294,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/42debffc-1ae6-42b4-ae40-7e52eabdc87b {"id":"42debffc-1ae6-42b4-ae40-7e52eabdc87b","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996101} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb5c8862 42debffc-1ae6-42b4-ae40-7e52eabdc87b /tidb/cdc/default/default/upstream/7365852066098678294 {"id":7365852066098678294,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/42debffc-1ae6-42b4-ae40-7e52eabdc87b {"id":"42debffc-1ae6-42b4-ae40-7e52eabdc87b","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996101} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb5c8862 42debffc-1ae6-42b4-ae40-7e52eabdc87b /tidb/cdc/default/default/upstream/7365852066098678294 {"id":7365852066098678294,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 12-th check, retry later public/default ++ date + echo '[Mon May 6 19:48:25 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:48:25 CST 2024] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.cli.3287.out cli changefeed create --start-ts=449575936179306497 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-resolve-lock-test-26595?protocol=canal-json&enable-tidb-extension=true' [Mon May 6 19:48:25 CST 2024] <<<<<< START Pulsar consumer in gc_safepoint case >>>>>> 0 Create changefeed successfully! ID: 011424d3-5b19-4eb2-b0d0-e9df144fa502 Info: {"upstream_id":7365852082963684363,"namespace":"default","id":"011424d3-5b19-4eb2-b0d0-e9df144fa502","sink_uri":"pulsar://127.0.0.1:6650/ticdc-simple-test-28323?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:48:25.710555999+08:00","start_ts":449575935965134849,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575935965134849,"checkpoint_ts":449575935965134849,"checkpoint_time":"2024-05-06 19:48:13.617"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... check diff failed 1-th time, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc) 3667975680 bytes in 7.20 secs (509756077 bytes/sec) [Pipeline] { [Pipeline] cache + set +x [Mon May 6 19:48:26 CST 2024] <<<<<< START Pulsar consumer in drop_many_tables case >>>>>> Create changefeed successfully! ID: 83ca7801-4ac5-40e1-b211-c101a71f8edf Info: {"upstream_id":7365852080212050206,"namespace":"default","id":"83ca7801-4ac5-40e1-b211-c101a71f8edf","sink_uri":"pulsar://127.0.0.1:6650/ticdc-resolve-lock-test-26595?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:48:26.344236729+08:00","start_ts":449575936179306497,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575936179306497,"checkpoint_ts":449575936179306497,"checkpoint_time":"2024-05-06 19:48:14.434"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:48:26 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/afff6dd3-7121-472e-8dec-81b96192be17 {"id":"afff6dd3-7121-472e-8dec-81b96192be17","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996103} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb5c04d4 afff6dd3-7121-472e-8dec-81b96192be17 /tidb/cdc/default/default/upstream/7365852064418707393 {"id":7365852064418707393,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/afff6dd3-7121-472e-8dec-81b96192be17 {"id":"afff6dd3-7121-472e-8dec-81b96192be17","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996103} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb5c04d4 afff6dd3-7121-472e-8dec-81b96192be17 /tidb/cdc/default/default/upstream/7365852064418707393 {"id":7365852064418707393,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/afff6dd3-7121-472e-8dec-81b96192be17 {"id":"afff6dd3-7121-472e-8dec-81b96192be17","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996103} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb5c04d4 afff6dd3-7121-472e-8dec-81b96192be17 /tidb/cdc/default/default/upstream/7365852064418707393 {"id":7365852064418707393,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ef49740001 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-fqtgg, pid:4700, start at 2024-05-06 19:48:25.821364647 +0800 CST m=+5.254664498 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:25.829 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:25.821 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:25.821 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table drop_tables.c not exists for 1-th check, retry later + set +x [Mon May 6 19:48:27 CST 2024] <<<<<< START Pulsar consumer in simple case >>>>>> succeed to verify meta placement rules table test.finish_mark not exists for 13-th check, retry later ERROR 1146 (42S02) at line 1: Table 'test.simple1' doesn't exist check data failed 1-th time, retry later [Mon May 6 19:48:27 CST 2024] <<<<<< START Pulsar consumer in processor_err_chan case >>>>>> check_changefeed_state http://127.0.0.1:2379 0a20f038-f254-4756-ad87-a38ea25a2daf normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=0a20f038-f254-4756-ad87-a38ea25a2daf + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 0a20f038-f254-4756-ad87-a38ea25a2daf -s + info='{ "upstream_id": 7365852064418707393, "namespace": "default", "id": "0a20f038-f254-4756-ad87-a38ea25a2daf", "state": "normal", "checkpoint_tso": 449575939349676039, "checkpoint_time": "2024-05-06 19:48:26.528", "error": null }' + echo '{ "upstream_id": 7365852064418707393, "namespace": "default", "id": "0a20f038-f254-4756-ad87-a38ea25a2daf", "state": "normal", "checkpoint_tso": 449575939349676039, "checkpoint_time": "2024-05-06 19:48:26.528", "error": null }' { "upstream_id": 7365852064418707393, "namespace": "default", "id": "0a20f038-f254-4756-ad87-a38ea25a2daf", "state": "normal", "checkpoint_tso": 449575939349676039, "checkpoint_time": "2024-05-06 19:48:26.528", "error": null } ++ echo '{' '"upstream_id":' 7365852064418707393, '"namespace":' '"default",' '"id":' '"0a20f038-f254-4756-ad87-a38ea25a2daf",' '"state":' '"normal",' '"checkpoint_tso":' 449575939349676039, '"checkpoint_time":' '"2024-05-06' '19:48:26.528",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7365852064418707393, '"namespace":' '"default",' '"id":' '"0a20f038-f254-4756-ad87-a38ea25a2daf",' '"state":' '"normal",' '"checkpoint_tso":' 449575939349676039, '"checkpoint_time":' '"2024-05-06' '19:48:26.528",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully + set +x [Mon May 6 19:48:27 CST 2024] <<<<<< START Pulsar consumer in resolve_lock case >>>>>> go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20240415145106-cd9c676e9ba4 go: downloading github.com/pingcap/kvproto v0.0.0-20240227073058-929ab83f9754 go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20240410110152-5fc42c9be2f5 go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading github.com/tikv/pd/client v0.0.0-20240322051414-fb9e2d561b6e go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20240409022718-714958ccd4d5 go: downloading github.com/BurntSushi/toml v1.3.2 go: downloading github.com/pingcap/tidb-tools v0.0.0-20240305021104-9f9bea84490b go: downloading go.uber.org/zap v1.27.0 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 check diff failed 1-th time, retry later go: downloading go.uber.org/atomic v1.11.0 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/pingcap/failpoint v0.0.0-20220801062533-2eaa32854a6c go: downloading google.golang.org/grpc v1.62.1 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/google/uuid v1.6.0 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/prometheus/client_golang v1.19.0 go: downloading github.com/golang/protobuf v1.5.4 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 go: downloading github.com/docker/go-units v0.5.0 go: downloading golang.org/x/sync v0.7.0 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/google/btree v1.1.2 go: downloading google.golang.org/protobuf v1.33.0 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/prometheus/common v0.52.2 go: downloading github.com/prometheus/procfs v0.13.0 check diff failed 2-th time, retry later go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading golang.org/x/exp v0.0.0-20240409090435-93d18d7e34b8 go: downloading golang.org/x/sys v0.19.0 go: downloading golang.org/x/net v0.24.0 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240401170217-c3f982113cda go: downloading golang.org/x/text v0.14.0 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ef49740001 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-fqtgg, pid:4700, start at 2024-05-06 19:48:25.821364647 +0800 CST m=+5.254664498 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:25.829 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:25.821 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:25.821 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36ef489c0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-fqtgg, pid:4782, start at 2024-05-06 19:48:25.800168804 +0800 CST m=+5.165357279 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:25.808 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:25.767 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:25.767 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/split_region/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/split_region/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/split_region/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/split_region/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/split_region/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table drop_tables.c not exists for 2-th check, retry later table test.finish_mark not exists for 14-th check, retry later TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 2-th time, retry later check diff successfully check_safepoint_forward http://127.0.0.1:2379 7365852066098678294 449575939948412934 449575938886729731 check diff failed 2-th time, retry later go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/pingcap/tipb v0.0.0-20240318032315-55a7867ddd50 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/gorilla/mux v1.8.0 go: downloading github.com/danjacques/gofslock v0.0.0-20240212154529-d899e02bfe22 go: downloading github.com/shirou/gopsutil/v3 v3.24.2 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/stretchr/testify v1.9.0 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/scalalang2/golang-fifo v0.1.5 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/otiai10/copy v1.2.0 go: downloading github.com/aws/aws-sdk-go v1.50.0 go: downloading github.com/tidwall/btree v1.7.0 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading golang.org/x/tools v0.20.0 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/vbauerster/mpb/v7 v7.5.3 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/fatih/color v1.16.0 go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading golang.org/x/term v0.19.0 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/klauspost/compress v1.17.8 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/spkg/bom v1.0.0 go: downloading golang.org/x/time v0.5.0 go: downloading github.com/xitongsys/parquet-go v1.6.0 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading github.com/spf13/cobra v1.8.0 go: downloading github.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/google/pprof v0.0.0-20240117000934-35fc243c5815 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading golang.org/x/oauth2 v0.18.0 go: downloading google.golang.org/api v0.170.0 go: downloading github.com/pingcap/badger v1.5.1-0.20230103063557-828f39b09b6d go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/Masterminds/semver v1.5.0 go: downloading k8s.io/api v0.28.6 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading cloud.google.com/go v0.112.2 go: downloading github.com/mattn/go-runewidth v0.0.15 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/jfcg/sixb v1.3.8 go: downloading github.com/emirpasic/gods v1.18.1 go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.1 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.1 go: downloading golang.org/x/crypto v0.22.0 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/golang/glog v1.2.0 go: downloading cloud.google.com/go/compute/metadata v0.2.3 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading cloud.google.com/go/compute v1.25.1 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/kr/pretty v0.3.1 go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.12.0 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/golang-jwt/jwt/v5 v5.2.0 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading github.com/golang-jwt/jwt v3.2.2+incompatible go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da run task successfully check_changefeed_state http://127.0.0.1:2379 cd1bb0a7-b47c-40b8-91cd-e299dd8ec850 stopped null + endpoints=http://127.0.0.1:2379 + changefeed_id=cd1bb0a7-b47c-40b8-91cd-e299dd8ec850 + expected_state=stopped + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c cd1bb0a7-b47c-40b8-91cd-e299dd8ec850 -s go: downloading github.com/go-logr/logr v1.4.1 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading go.opentelemetry.io/otel/metric v1.24.0 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.cli.6132.out cli tso query --pd=http://127.0.0.1:2379 + info='{ "upstream_id": 7365852066098678294, "namespace": "default", "id": "cd1bb0a7-b47c-40b8-91cd-e299dd8ec850", "state": "stopped", "checkpoint_tso": 449575940472700935, "checkpoint_time": "2024-05-06 19:48:30.812", "error": null }' + echo '{ "upstream_id": 7365852066098678294, "namespace": "default", "id": "cd1bb0a7-b47c-40b8-91cd-e299dd8ec850", "state": "stopped", "checkpoint_tso": 449575940472700935, "checkpoint_time": "2024-05-06 19:48:30.812", "error": null }' { "upstream_id": 7365852066098678294, "namespace": "default", "id": "cd1bb0a7-b47c-40b8-91cd-e299dd8ec850", "state": "stopped", "checkpoint_tso": 449575940472700935, "checkpoint_time": "2024-05-06 19:48:30.812", "error": null } ++ echo '{' '"upstream_id":' 7365852066098678294, '"namespace":' '"default",' '"id":' '"cd1bb0a7-b47c-40b8-91cd-e299dd8ec850",' '"state":' '"stopped",' '"checkpoint_tso":' 449575940472700935, '"checkpoint_time":' '"2024-05-06' '19:48:30.812",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7365852066098678294, '"namespace":' '"default",' '"id":' '"cd1bb0a7-b47c-40b8-91cd-e299dd8ec850",' '"state":' '"stopped",' '"checkpoint_tso":' 449575940472700935, '"checkpoint_time":' '"2024-05-06' '19:48:30.812",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_equal http://127.0.0.1:2379 7365852066098678294 go: downloading github.com/DataDog/zstd v1.5.5 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 table drop_tables.c not exists for 3-th check, retry later table test.finish_mark not exists for 15-th check, retry later go: downloading k8s.io/apimachinery v0.28.6 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading k8s.io/klog/v2 v2.120.1 go: downloading github.com/google/gofuzz v1.2.0 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 3-th time, retry later check diff failed 3-th time, retry later + set +x + tso='449575940579917826 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575940579917826 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:48:32 CST 2024] <<<<<< START cdc server in split_region case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.61816183.out server --log-file /tmp/tidb_cdc_test/split_region/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/split_region/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/changefeed_error/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table drop_tables.c not exists for 4-th check, retry later table test.finish_mark not exists for 16-th check, retry later check data successfully wait process cdc.test exit for 1-th time... check diff successfully run task successfully start tidb cluster in /tmp/tidb_cdc_test/changefeed_error Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 1-th time... check_changefeed_state http://127.0.0.1:2379 cd1bb0a7-b47c-40b8-91cd-e299dd8ec850 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=cd1bb0a7-b47c-40b8-91cd-e299dd8ec850 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c cd1bb0a7-b47c-40b8-91cd-e299dd8ec850 -s go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading github.com/google/s2a-go v0.1.7 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading github.com/felixge/httpsnoop v1.0.4 go: downloading github.com/jmespath/go-jmespath v0.4.0 wait process cdc.test exit for 3-th time... process cdc.test already exit [Mon May 6 19:48:35 CST 2024] <<<<<< run test case simple success! >>>>>> + info='{ "upstream_id": 7365852066098678294, "namespace": "default", "id": "cd1bb0a7-b47c-40b8-91cd-e299dd8ec850", "state": "normal", "checkpoint_tso": 449575940472700935, "checkpoint_time": "2024-05-06 19:48:30.812", "error": null }' + echo '{ "upstream_id": 7365852066098678294, "namespace": "default", "id": "cd1bb0a7-b47c-40b8-91cd-e299dd8ec850", "state": "normal", "checkpoint_tso": 449575940472700935, "checkpoint_time": "2024-05-06 19:48:30.812", "error": null }' { "upstream_id": 7365852066098678294, "namespace": "default", "id": "cd1bb0a7-b47c-40b8-91cd-e299dd8ec850", "state": "normal", "checkpoint_tso": 449575940472700935, "checkpoint_time": "2024-05-06 19:48:30.812", "error": null } ++ echo '{' '"upstream_id":' 7365852066098678294, '"namespace":' '"default",' '"id":' '"cd1bb0a7-b47c-40b8-91cd-e299dd8ec850",' '"state":' '"normal",' '"checkpoint_tso":' 449575940472700935, '"checkpoint_time":' '"2024-05-06' '19:48:30.812",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7365852066098678294, '"namespace":' '"default",' '"id":' '"cd1bb0a7-b47c-40b8-91cd-e299dd8ec850",' '"state":' '"normal",' '"checkpoint_tso":' 449575940472700935, '"checkpoint_time":' '"2024-05-06' '19:48:30.812",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_forward http://127.0.0.1:2379 7365852066098678294 449575940472700934 449575940472700935 table drop_tables.c not exists for 5-th check, retry later wait process cdc.test exit for 2-th time... table test.finish_mark exists check diff successfully Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc) 3667975680 bytes in 7.65 secs (479386518 bytes/sec) [Pipeline] { [Pipeline] cache wait process cdc.test exit for 3-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:48:36 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/649b12bc-6ba5-4c89-a55e-d5ab6663e403 {"id":"649b12bc-6ba5-4c89-a55e-d5ab6663e403","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996113} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbba60158 649b12bc-6ba5-4c89-a55e-d5ab6663e403 /tidb/cdc/default/default/upstream/7365852149827278214 {"id":7365852149827278214,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/649b12bc-6ba5-4c89-a55e-d5ab6663e403 {"id":"649b12bc-6ba5-4c89-a55e-d5ab6663e403","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996113} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbba60158 649b12bc-6ba5-4c89-a55e-d5ab6663e403 /tidb/cdc/default/default/upstream/7365852149827278214 {"id":7365852149827278214,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/649b12bc-6ba5-4c89-a55e-d5ab6663e403 {"id":"649b12bc-6ba5-4c89-a55e-d5ab6663e403","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996113} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbba60158 649b12bc-6ba5-4c89-a55e-d5ab6663e403 /tidb/cdc/default/default/upstream/7365852149827278214 {"id":7365852149827278214,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:48:36 CST 2024] <<<<<< START pulsar cluster in split_region case >>>>>>' [Mon May 6 19:48:36 CST 2024] <<<<<< START pulsar cluster in split_region case >>>>>> + workdir=/tmp/tidb_cdc_test/split_region + cluster_type=oauth + cd /tmp/tidb_cdc_test/split_region + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/split_region/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/split_region/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/split_region/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/split_region/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/split_region/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/split_region/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/split_region/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/split_region/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/split_region/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/split_region/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/split_region/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/split_region/pulsar_standalone.conf + pulsar_port=6650 + '[' oauth == mtls ']' + '[' oauth == oauth ']' + echo 'Waiting for oauth2 server to be ready...' Waiting for oauth2 server to be ready... + i=0 + nc -z localhost 9096 + oauth2-server + gen_oauth_config + echo ' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' + echo ' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/split_region/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/split_region/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ date + echo '[Mon May 6 19:48:36 CST 2024] <<<<<< START pulsar cluster in oauth mode in split_region case >>>>>>' [Mon May 6 19:48:36 CST 2024] <<<<<< START pulsar cluster in oauth mode in split_region case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/split_region/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/split_region/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/split_region/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:48:36 CST 2024] <<<<<< run test case processor_err_chan success! >>>>>> run task successfully check_changefeed_state http://127.0.0.1:2379 cd1bb0a7-b47c-40b8-91cd-e299dd8ec850 stopped null + endpoints=http://127.0.0.1:2379 + changefeed_id=cd1bb0a7-b47c-40b8-91cd-e299dd8ec850 + expected_state=stopped + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c cd1bb0a7-b47c-40b8-91cd-e299dd8ec850 -s + info='{ "upstream_id": 7365852066098678294, "namespace": "default", "id": "cd1bb0a7-b47c-40b8-91cd-e299dd8ec850", "state": "stopped", "checkpoint_tso": 449575940472700935, "checkpoint_time": "2024-05-06 19:48:30.812", "error": null }' + echo '{ "upstream_id": 7365852066098678294, "namespace": "default", "id": "cd1bb0a7-b47c-40b8-91cd-e299dd8ec850", "state": "stopped", "checkpoint_tso": 449575940472700935, "checkpoint_time": "2024-05-06 19:48:30.812", "error": null }' { "upstream_id": 7365852066098678294, "namespace": "default", "id": "cd1bb0a7-b47c-40b8-91cd-e299dd8ec850", "state": "stopped", "checkpoint_tso": 449575940472700935, "checkpoint_time": "2024-05-06 19:48:30.812", "error": null } ++ echo '{' '"upstream_id":' 7365852066098678294, '"namespace":' '"default",' '"id":' '"cd1bb0a7-b47c-40b8-91cd-e299dd8ec850",' '"state":' '"stopped",' '"checkpoint_tso":' 449575940472700935, '"checkpoint_time":' '"2024-05-06' '19:48:30.812",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7365852066098678294, '"namespace":' '"default",' '"id":' '"cd1bb0a7-b47c-40b8-91cd-e299dd8ec850",' '"state":' '"stopped",' '"checkpoint_tso":' 449575940472700935, '"checkpoint_time":' '"2024-05-06' '19:48:30.812",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_changefeed_state http://127.0.0.1:2379 52c4f030-eb81-41e4-9d29-03e950ddf5a4 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=52c4f030-eb81-41e4-9d29-03e950ddf5a4 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 52c4f030-eb81-41e4-9d29-03e950ddf5a4 -s + info='{ "upstream_id": 7365852066098678294, "namespace": "default", "id": "52c4f030-eb81-41e4-9d29-03e950ddf5a4", "state": "normal", "checkpoint_tso": 449575942019350529, "checkpoint_time": "2024-05-06 19:48:36.712", "error": null }' + echo '{ "upstream_id": 7365852066098678294, "namespace": "default", "id": "52c4f030-eb81-41e4-9d29-03e950ddf5a4", "state": "normal", "checkpoint_tso": 449575942019350529, "checkpoint_time": "2024-05-06 19:48:36.712", "error": null }' { "upstream_id": 7365852066098678294, "namespace": "default", "id": "52c4f030-eb81-41e4-9d29-03e950ddf5a4", "state": "normal", "checkpoint_tso": 449575942019350529, "checkpoint_time": "2024-05-06 19:48:36.712", "error": null } ++ echo '{' '"upstream_id":' 7365852066098678294, '"namespace":' '"default",' '"id":' '"52c4f030-eb81-41e4-9d29-03e950ddf5a4",' '"state":' '"normal",' '"checkpoint_tso":' 449575942019350529, '"checkpoint_time":' '"2024-05-06' '19:48:36.712",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7365852066098678294, '"namespace":' '"default",' '"id":' '"52c4f030-eb81-41e4-9d29-03e950ddf5a4",' '"state":' '"normal",' '"checkpoint_tso":' 449575942019350529, '"checkpoint_time":' '"2024-05-06' '19:48:36.712",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_equal http://127.0.0.1:2379 7365852066098678294 table drop_tables.c exists check diff successfully Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release wait process cdc.test exit for 1-th time... + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:48:39 CST 2024] <<<<<< run test case drop_many_tables success! >>>>>> Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) run task successfully Changefeed remove successfully. ID: cd1bb0a7-b47c-40b8-91cd-e299dd8ec850 CheckpointTs: 449575940472700935 SinkURI: pulsar://127.0.0.1:6650/ticdc-gc-safepoint-5425?protocol=canal-json&enable-tidb-extension=true check_safepoint_forward http://127.0.0.1:2379 7365852066098678294 449575942019350528 449575942019350529 449575940472700935 + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public table test.finish_mark not exists for 1-th check, retry later wait process 5811 exit for 1-th time... wait process 5811 exit for 2-th time... wait process 5811 exit for 3-th time... wait process 5811 exit for 4-th time... wait process 5811 exit for 5-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process 5811 exit for 6-th time... run task successfully /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (5811) - No such process wait process 5811 exit for 7-th time... process 5811 already exit [Mon May 6 19:48:41 CST 2024] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.64776479.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table test.finish_mark not exists for 2-th check, retry later Changefeed remove successfully. ID: 52c4f030-eb81-41e4-9d29-03e950ddf5a4 CheckpointTs: 449575943094403080 SinkURI: pulsar://127.0.0.1:6650/ticdc-gc-safepoint-5425?protocol=canal-json&enable-tidb-extension=true check_safepoint_cleared http://127.0.0.1:2379 7365852066098678294 run task successfully Tenant not found Reason: Tenant not found wait process cdc.test exit for 1-th time... + i=1 + '[' 1 -gt 20 ']' + sleep 2 wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 3-th time... table test.finish_mark not exists for 3-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:48:44 CST 2024] <<<<<< run test case gc_safepoint success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > + /usr/local/pulsar/bin/pulsar-admin namespaces list public ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 4-th check, retry later public/default VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f06e8c0008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jzh76, pid:4523, start at 2024-05-06 19:48:44.587760181 +0800 CST m=+5.262887810 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:44.595 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:44.579 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:44.579 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f06e8c0008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jzh76, pid:4523, start at 2024-05-06 19:48:44.587760181 +0800 CST m=+5.262887810 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:44.595 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:44.579 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:44.579 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f06f480010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jzh76, pid:4601, start at 2024-05-06 19:48:44.646784839 +0800 CST m=+5.272162888 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:50:44.654 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:48:44.626 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:38:44.626 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_error/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_error/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ++ date + echo '[Mon May 6 19:48:47 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:48:47 CST 2024] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.cli.6722.out cli changefeed create --start-ts=449575940579917826 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-split-region-test-21153?protocol=canal-json&enable-tidb-extension=true' --config /tmp/tidb_cdc_test/split_region/pulsar_test.toml table test.finish_mark not exists for 5-th check, retry later < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:48:48 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7365852043233991414 Namespace:default ID:ddl-manager SinkURI:pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-12528?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:48:21.572726011 +0800 CST StartTs:449575937039663106 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003bf2240 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575938048917506} {CheckpointTs:449575938835349528 MinTableBarrierTs:449575938835349528 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4a35e301-4d0a-460d-9f3b-ea54ba25d782 {"id":"4a35e301-4d0a-460d-9f3b-ea54ba25d782","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996122} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb46e0f3 4a35e301-4d0a-460d-9f3b-ea54ba25d782 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7365852043233991414,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-12528?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:48:21.572726011+08:00","start-ts":449575937039663106,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575938048917506} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":449575938835349528,"min-table-barrier-ts":449575938835349528,"admin-job-type":0} /tidb/cdc/default/default/task/position/4a35e301-4d0a-460d-9f3b-ea54ba25d782/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852043233991414 {"id":7365852043233991414,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7365852043233991414 Namespace:default ID:ddl-manager SinkURI:pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-12528?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:48:21.572726011 +0800 CST StartTs:449575937039663106 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003bf2240 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575938048917506} {CheckpointTs:449575938835349528 MinTableBarrierTs:449575938835349528 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4a35e301-4d0a-460d-9f3b-ea54ba25d782 {"id":"4a35e301-4d0a-460d-9f3b-ea54ba25d782","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996122} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb46e0f3 4a35e301-4d0a-460d-9f3b-ea54ba25d782 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7365852043233991414,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-12528?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:48:21.572726011+08:00","start-ts":449575937039663106,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575938048917506} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":449575938835349528,"min-table-barrier-ts":449575938835349528,"admin-job-type":0} /tidb/cdc/default/default/task/position/4a35e301-4d0a-460d-9f3b-ea54ba25d782/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852043233991414 + grep -q 'failed to get info:' {"id":7365852043233991414,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7365852043233991414 Namespace:default ID:ddl-manager SinkURI:pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-12528?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:48:21.572726011 +0800 CST StartTs:449575937039663106 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003bf2240 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575938048917506} {CheckpointTs:449575938835349528 MinTableBarrierTs:449575938835349528 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4a35e301-4d0a-460d-9f3b-ea54ba25d782 {"id":"4a35e301-4d0a-460d-9f3b-ea54ba25d782","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996122} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb46e0f3 4a35e301-4d0a-460d-9f3b-ea54ba25d782 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7365852043233991414,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-12528?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:48:21.572726011+08:00","start-ts":449575937039663106,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575938048917506} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":449575938835349528,"min-table-barrier-ts":449575938835349528,"admin-job-type":0} /tidb/cdc/default/default/task/position/4a35e301-4d0a-460d-9f3b-ea54ba25d782/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852043233991414 + grep -q 'etcd info' {"id":7365852043233991414,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x [Mon May 6 19:48:48 CST 2024] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.65376539.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/changefeed_reconstruct/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... Create changefeed successfully! ID: d48465fb-4796-4fb0-8631-8dcc2c74af4f Info: {"upstream_id":7365852149827278214,"namespace":"default","id":"d48465fb-4796-4fb0-8631-8dcc2c74af4f","sink_uri":"pulsar://127.0.0.1:6650/ticdc-split-region-test-21153?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:48:48.927812833+08:00","start_ts":449575940579917826,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30,"oauth2":{"oauth2-issuer-url":"http://localhost:9096","oauth2-audience":"cdc-api-uri","oauth2-private-key":"/tmp/tidb_cdc_test/split_region/credential.json","oauth2-client-id":"1234"}},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575940579917826,"checkpoint_ts":449575940579917826,"checkpoint_time":"2024-05-06 19:48:31.221"} PASS =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/processor_stop_delay/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... coverage: 2.7% of statements in github.com/pingcap/tiflow/... + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5888.out cli tso query --pd=http://127.0.0.1:2379 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/cdc_server_tips/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table test.finish_mark not exists for 6-th check, retry later + set +x [Mon May 6 19:48:50 CST 2024] <<<<<< START Pulsar consumer in split_region case >>>>>> table split_region.test1 not exists for 1-th check, retry later + set +x + tso='449575945484369921 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575945484369921 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:48:51 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7365852043233991414 Namespace:default ID:ddl-manager SinkURI:pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-12528?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:48:21.572726011 +0800 CST StartTs:449575937039663106 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003bf2240 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575938048917506} {CheckpointTs:449575938835349528 MinTableBarrierTs:449575938835349528 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4a35e301-4d0a-460d-9f3b-ea54ba25d782 {"id":"4a35e301-4d0a-460d-9f3b-ea54ba25d782","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996122} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb46e0f3 4a35e301-4d0a-460d-9f3b-ea54ba25d782 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7365852043233991414,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-12528?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:48:21.572726011+08:00","start-ts":449575937039663106,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575938048917506} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":449575938966421532,"min-table-barrier-ts":449575938966421532,"admin-job-type":0} /tidb/cdc/default/default/task/position/4a35e301-4d0a-460d-9f3b-ea54ba25d782/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852043233991414 {"id":7365852043233991414,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7365852043233991414 Namespace:default ID:ddl-manager SinkURI:pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-12528?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:48:21.572726011 +0800 CST StartTs:449575937039663106 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003bf2240 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575938048917506} {CheckpointTs:449575938835349528 MinTableBarrierTs:449575938835349528 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4a35e301-4d0a-460d-9f3b-ea54ba25d782 {"id":"4a35e301-4d0a-460d-9f3b-ea54ba25d782","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996122} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb46e0f3 4a35e301-4d0a-460d-9f3b-ea54ba25d782 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7365852043233991414,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-12528?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:48:21.572726011+08:00","start-ts":449575937039663106,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575938048917506} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":449575938966421532,"min-table-barrier-ts":449575938966421532,"admin-job-type":0} /tidb/cdc/default/default/task/position/4a35e301-4d0a-460d-9f3b-ea54ba25d782/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852043233991414 + grep -q 'failed to get info:' {"id":7365852043233991414,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7365852043233991414 Namespace:default ID:ddl-manager SinkURI:pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-12528?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:48:21.572726011 +0800 CST StartTs:449575937039663106 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003bf2240 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575938048917506} {CheckpointTs:449575938835349528 MinTableBarrierTs:449575938835349528 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4a35e301-4d0a-460d-9f3b-ea54ba25d782 {"id":"4a35e301-4d0a-460d-9f3b-ea54ba25d782","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996122} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbb46e0f3 4a35e301-4d0a-460d-9f3b-ea54ba25d782 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7365852043233991414,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"pulsar://127.0.0.1:6650/ticdc-ddl-mamager-test-12528?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:48:21.572726011+08:00","start-ts":449575937039663106,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575938048917506} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":449575938966421532,"min-table-barrier-ts":449575938966421532,"admin-job-type":0} /tidb/cdc/default/default/task/position/4a35e301-4d0a-460d-9f3b-ea54ba25d782/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852043233991414 {"id":7365852043233991414,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x table ddl_manager.finish_mark not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/changefeed_reconstruct Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... table test.finish_mark not exists for 7-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/cdc_server_tips Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/processor_stop_delay Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... table split_region.test1 exists table split_region.test2 not exists for 1-th check, retry later ***************** properties ***************** "threadcount"="4" "workload"="core" "mysql.host"="127.0.0.1" "updateproportion"="0" "operationcount"="0" "recordcount"="20" "mysql.db"="changefeed_error" "insertproportion"="0" "mysql.port"="4000" "readproportion"="0" "mysql.user"="root" "scanproportion"="0" "dotransactions"="false" "requestdistribution"="uniform" "readallfields"="true" ********************************************** Run finished, takes 18.018272ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2233.1, Avg(us): 3352, Min(us): 1365, Max(us): 9583, 95th(us): 10000, 99th(us): 10000 [Mon May 6 19:48:52 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/NewChangefeedNoRetryError=1*return(true)' + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.59525954.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 2-th check, retry later Verifying downstream PD is started... table test.finish_mark not exists for 8-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/changefeed_pause_resume/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table split_region.test2 exists check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 3-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:48:55 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c73e6550-2b13-4366-a73e-72990490cfa1 {"id":"c73e6550-2b13-4366-a73e-72990490cfa1","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996132} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbbf52251 c73e6550-2b13-4366-a73e-72990490cfa1 /tidb/cdc/default/default/upstream/7365852232033146232 {"id":7365852232033146232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c73e6550-2b13-4366-a73e-72990490cfa1 {"id":"c73e6550-2b13-4366-a73e-72990490cfa1","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996132} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbbf52251 c73e6550-2b13-4366-a73e-72990490cfa1 /tidb/cdc/default/default/upstream/7365852232033146232 {"id":7365852232033146232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c73e6550-2b13-4366-a73e-72990490cfa1 {"id":"c73e6550-2b13-4366-a73e-72990490cfa1","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996132} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbbf52251 c73e6550-2b13-4366-a73e-72990490cfa1 /tidb/cdc/default/default/upstream/7365852232033146232 {"id":7365852232033146232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:48:55 CST 2024] <<<<<< START pulsar cluster in changefeed_error case >>>>>>' [Mon May 6 19:48:55 CST 2024] <<<<<< START pulsar cluster in changefeed_error case >>>>>> + workdir=/tmp/tidb_cdc_test/changefeed_error + cluster_type=normal + cd /tmp/tidb_cdc_test/changefeed_error + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_error/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/changefeed_error/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/changefeed_error/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_error/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/changefeed_error/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/changefeed_error/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/changefeed_error/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/changefeed_error/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_error/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/changefeed_error/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/changefeed_error/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/changefeed_error/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:48:55 CST 2024] <<<<<< START pulsar cluster in normal mode in changefeed_error case >>>>>>' [Mon May 6 19:48:55 CST 2024] <<<<<< START pulsar cluster in normal mode in changefeed_error case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/changefeed_error/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/changefeed_error/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/changefeed_error/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 table test.finish_mark exists check diff successfully Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release wait process cdc.test exit for 1-th time... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff successfully wait process cdc.test exit for 2-th time... Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 4-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/changefeed_pause_resume Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... wait process cdc.test exit for 3-th time... Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc) 3667975680 bytes in 19.12 secs (191870840 bytes/sec) [Pipeline] { [Pipeline] cache + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 Starting Upstream TiDB... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:48:57 CST 2024] <<<<<< run test case canal_json_content_compatible success! >>>>>> Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 5-th check, retry later + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 6-th check, retry later public/default Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ++ date + echo '[Mon May 6 19:49:01 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:49:01 CST 2024] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6414.out cli changefeed create --start-ts=449575945484369921 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-sink-retry-test-4702?protocol=canal-json&enable-tidb-extension=true' -c changefeed-error ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Create changefeed successfully! ID: changefeed-error Info: {"upstream_id":7365852232033146232,"namespace":"default","id":"changefeed-error","sink_uri":"pulsar://127.0.0.1:6650/ticdc-sink-retry-test-4702?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:49:02.802513524+08:00","start_ts":449575945484369921,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575945484369921,"checkpoint_ts":449575945484369921,"checkpoint_time":"2024-05-06 19:48:49.930"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 7-th check, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x [Mon May 6 19:49:04 CST 2024] <<<<<< START Pulsar consumer in changefeed_error case >>>>>> check_changefeed_state http://127.0.0.1:2379 changefeed-error failed [CDC:ErrStartTsBeforeGC] + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error + expected_state=failed + error_msg='[CDC:ErrStartTsBeforeGC]' + tls_dir='[CDC:ErrStartTsBeforeGC]' + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error -s VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f18b240012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh, pid:4499, start at 2024-05-06 19:49:02.810703173 +0800 CST m=+6.137649718 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:02.816 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:02.793 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:02.793 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f18b240012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh, pid:4499, start at 2024-05-06 19:49:02.810703173 +0800 CST m=+6.137649718 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:02.816 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:02.793 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:02.793 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f17dbc0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh, pid:4577, start at 2024-05-06 19:49:01.984540005 +0800 CST m=+5.262129727 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:01.993 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:01.985 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:01.985 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + info='{ "upstream_id": 7365852232033146232, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 449575945484369921, "checkpoint_time": "2024-05-06 19:48:49.930", "error": { "time": "2024-05-06T19:49:02.889723581+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 449575945484369621 is earlier than or equal to GC safepoint at 449575945484369921" } }' + echo '{ "upstream_id": 7365852232033146232, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 449575945484369921, "checkpoint_time": "2024-05-06 19:48:49.930", "error": { "time": "2024-05-06T19:49:02.889723581+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 449575945484369621 is earlier than or equal to GC safepoint at 449575945484369921" } }' { "upstream_id": 7365852232033146232, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 449575945484369921, "checkpoint_time": "2024-05-06 19:48:49.930", "error": { "time": "2024-05-06T19:49:02.889723581+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 449575945484369621 is earlier than or equal to GC safepoint at 449575945484369921" } } ++ echo '{' '"upstream_id":' 7365852232033146232, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"failed",' '"checkpoint_tso":' 449575945484369921, '"checkpoint_time":' '"2024-05-06' '19:48:49.930",' '"error":' '{' '"time":' '"2024-05-06T19:49:02.889723581+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]fail' to create or maintain changefeed because start-ts 449575945484369621 is earlier than or equal to GC safepoint at '449575945484369921"' '}' '}' ++ jq -r .state + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ jq -r .error.message ++ echo '{' '"upstream_id":' 7365852232033146232, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"failed",' '"checkpoint_tso":' 449575945484369921, '"checkpoint_time":' '"2024-05-06' '19:48:49.930",' '"error":' '{' '"time":' '"2024-05-06T19:49:02.889723581+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]fail' to create or maintain changefeed because start-ts 449575945484369621 is earlier than or equal to GC safepoint at '449575945484369921"' '}' '}' + message='[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 449575945484369621 is earlier than or equal to GC safepoint at 449575945484369921' + [[ ! [CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 449575945484369621 is earlier than or equal to GC safepoint at 449575945484369921 =~ \[CDC:ErrStartTsBeforeGC] ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6548.out cli changefeed resume -c changefeed-error VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f194c0000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6, pid:4713, start at 2024-05-06 19:49:03.41603014 +0800 CST m=+5.165284870 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:03.422 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:03.408 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:03.408 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f194c0000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6, pid:4713, start at 2024-05-06 19:49:03.41603014 +0800 CST m=+5.165284870 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:03.422 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:03.408 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:03.408 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f194b00014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6, pid:4796, start at 2024-05-06 19:49:03.443730005 +0800 CST m=+5.137742761 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:03.450 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:03.454 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:03.454 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f199b40012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-k03d6, pid:7541, start at 2024-05-06 19:49:03.760782499 +0800 CST m=+5.296197149 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:03.767 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:03.725 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:03.725 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) PASS coverage: 2.3% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 8-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.cli.5921.out cli tso query --pd=http://127.0.0.1:2379 ++ date + echo '[Mon May 6 19:49:06 CST 2024] <<<<<< START pulsar cluster in changefeed_reconstruct case >>>>>>' [Mon May 6 19:49:06 CST 2024] <<<<<< START pulsar cluster in changefeed_reconstruct case >>>>>> + workdir=/tmp/tidb_cdc_test/changefeed_reconstruct + cluster_type=normal + cd /tmp/tidb_cdc_test/changefeed_reconstruct + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_reconstruct/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/changefeed_reconstruct/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/changefeed_reconstruct/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_reconstruct/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/changefeed_reconstruct/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/changefeed_reconstruct/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/changefeed_reconstruct/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/changefeed_reconstruct/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_reconstruct/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/changefeed_reconstruct/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/changefeed_reconstruct/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/changefeed_reconstruct/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:49:06 CST 2024] <<<<<< START pulsar cluster in normal mode in changefeed_reconstruct case >>>>>>' [Mon May 6 19:49:06 CST 2024] <<<<<< START pulsar cluster in normal mode in changefeed_reconstruct case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/changefeed_reconstruct/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/changefeed_reconstruct/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/changefeed_reconstruct/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 + set +x table changefeed_error.usertable not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f199b40012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-k03d6, pid:7541, start at 2024-05-06 19:49:03.760782499 +0800 CST m=+5.296197149 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:03.767 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:03.725 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:03.725 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f19b2c000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-k03d6, pid:7622, start at 2024-05-06 19:49:03.82978025 +0800 CST m=+5.312453323 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:03.836 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:03.819 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:03.819 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table ddl_manager.finish_mark not exists for 9-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + tso='449575949800046593 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575949800046593 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/multi_topics/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:49:07 CST 2024] <<<<<< run test case multi_topics success! >>>>>> + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 table changefeed_error.usertable exists try a VALID cdc server command [Mon May 6 19:49:08 CST 2024] <<<<<< START cdc server in cdc_server_tips case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.59615963.out server --log-file /tmp/tidb_cdc_test/cdc_server_tips/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cdc_server_tips/cdc_data --cluster-id default + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ++ date + echo '[Mon May 6 19:49:08 CST 2024] <<<<<< START pulsar cluster in processor_stop_delay case >>>>>>' [Mon May 6 19:49:08 CST 2024] <<<<<< START pulsar cluster in processor_stop_delay case >>>>>> + workdir=/tmp/tidb_cdc_test/processor_stop_delay + cluster_type=normal + cd /tmp/tidb_cdc_test/processor_stop_delay + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/processor_stop_delay/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/processor_stop_delay/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/processor_stop_delay/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/processor_stop_delay/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/processor_stop_delay/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/processor_stop_delay/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/processor_stop_delay/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/processor_stop_delay/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/processor_stop_delay/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/processor_stop_delay/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/processor_stop_delay/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/processor_stop_delay/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:49:08 CST 2024] <<<<<< START pulsar cluster in normal mode in processor_stop_delay case >>>>>>' [Mon May 6 19:49:08 CST 2024] <<<<<< START pulsar cluster in normal mode in processor_stop_delay case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/processor_stop_delay/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/processor_stop_delay/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/processor_stop_delay/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 check diff failed 1-th time, retry later check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 10-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f1dec40013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb, pid:5157, start at 2024-05-06 19:49:08.174838588 +0800 CST m=+5.166450637 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:08.183 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:08.145 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:08.145 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f1dec40013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb, pid:5157, start at 2024-05-06 19:49:08.174838588 +0800 CST m=+5.166450637 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:08.183 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:08.145 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:08.145 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f1e114000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb, pid:5242, start at 2024-05-06 19:49:08.305073096 +0800 CST m=+5.249206906 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:51:08.311 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:49:08.293 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:39:08.293 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff successfully ***************** properties ***************** "threadcount"="4" "readallfields"="true" "requestdistribution"="uniform" "recordcount"="20" "workload"="core" "mysql.host"="127.0.0.1" "dotransactions"="false" "mysql.port"="4000" "scanproportion"="0" "updateproportion"="0" "readproportion"="0" "insertproportion"="0" "mysql.user"="root" "operationcount"="0" "mysql.db"="changefeed_error" ********************************************** Run finished, takes 4.254085ms INSERT - Takes(s): 0.0, Count: 20, OPS: 6962.1, Avg(us): 739, Min(us): 428, Max(us): 1802, 95th(us): 2000, 99th(us): 2000 check diff successfully check diff successfully {"id":"c73e6550-2b13-4366-a73e-72990490cfa1","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996132} check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/c73e6550-2b13-4366-a73e-72990490cfa1 + [[ /tidb/cdc/default/__cdc_meta__/capture/c73e6550-2b13-4366-a73e-72990490cfa1 =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/c73e6550-2b13-4366-a73e-72990490cfa1' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/c73e6550-2b13-4366-a73e-72990490cfa1 + echo 'check failed' check failed + exit 1 run task failed 1-th time, retry later + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public wait process cdc.test exit for 1-th time... table ddl_manager.finish_mark not exists for 11-th check, retry later ++ date + echo '[Mon May 6 19:49:11 CST 2024] <<<<<< START pulsar cluster in changefeed_pause_resume case >>>>>>' [Mon May 6 19:49:11 CST 2024] <<<<<< START pulsar cluster in changefeed_pause_resume case >>>>>> + workdir=/tmp/tidb_cdc_test/changefeed_pause_resume + cluster_type=normal + cd /tmp/tidb_cdc_test/changefeed_pause_resume + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/avro_basic/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:49:11 CST 2024] <<<<<< run test case avro_basic success! >>>>>> + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_pause_resume/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/changefeed_pause_resume/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/changefeed_pause_resume/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_pause_resume/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/changefeed_pause_resume/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/changefeed_pause_resume/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/changefeed_pause_resume/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/changefeed_pause_resume/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_pause_resume/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/changefeed_pause_resume/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/changefeed_pause_resume/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/changefeed_pause_resume/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:49:11 CST 2024] <<<<<< START pulsar cluster in normal mode in changefeed_pause_resume case >>>>>>' [Mon May 6 19:49:11 CST 2024] <<<<<< START pulsar cluster in normal mode in changefeed_pause_resume case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/changefeed_pause_resume/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/changefeed_pause_resume/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/changefeed_pause_resume/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:49:11 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7855eb32-2a10-4447-92eb-ef0ed18bd7ff {"id":"7855eb32-2a10-4447-92eb-ef0ed18bd7ff","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996148} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbc38c648 7855eb32-2a10-4447-92eb-ef0ed18bd7ff /tidb/cdc/default/default/upstream/7365852303211969554 {"id":7365852303211969554,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7855eb32-2a10-4447-92eb-ef0ed18bd7ff {"id":"7855eb32-2a10-4447-92eb-ef0ed18bd7ff","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996148} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbc38c648 7855eb32-2a10-4447-92eb-ef0ed18bd7ff /tidb/cdc/default/default/upstream/7365852303211969554 {"id":7365852303211969554,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7855eb32-2a10-4447-92eb-ef0ed18bd7ff {"id":"7855eb32-2a10-4447-92eb-ef0ed18bd7ff","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996148} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbc38c648 7855eb32-2a10-4447-92eb-ef0ed18bd7ff /tidb/cdc/default/default/upstream/7365852303211969554 {"id":7365852303211969554,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/c73e6550-2b13-4366-a73e-72990490cfa1 + [[ /tidb/cdc/default/__cdc_meta__/capture/c73e6550-2b13-4366-a73e-72990490cfa1 =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/c73e6550-2b13-4366-a73e-72990490cfa1' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/c73e6550-2b13-4366-a73e-72990490cfa1 + echo 'check failed' check failed + exit 1 run task failed 2-th time, retry later public/default table ddl_manager.finish_mark not exists for 12-th check, retry later + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 ++ date + echo '[Mon May 6 19:49:13 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:49:13 CST 2024] <<<<<< pulsar is ready >>>>>> [Mon May 6 19:49:13 CST 2024] <<<<<< START cdc server in changefeed_reconstruct case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_reconstruct.64666468.out server --log-file /tmp/tidb_cdc_test/changefeed_reconstruct/cdcserver1.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_reconstruct/cdc_dataserver1 --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 2-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/canal_json_handle_key_only/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:49:14 CST 2024] <<<<<< run test case canal_json_handle_key_only success! >>>>>> cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Mon May 6 19:49:14 CST 2024] <<<<<< run test case split_region success! >>>>>> public/default ++ date + echo '[Mon May 6 19:49:15 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:49:15 CST 2024] <<<<<< pulsar is ready >>>>>> [Mon May 6 19:49:15 CST 2024] <<<<<< START cdc server in processor_stop_delay case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorStopDelay=1*sleep(10000)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_stop_delay.92989300.out server --log-file /tmp/tidb_cdc_test/processor_stop_delay/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_stop_delay/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 13-th check, retry later + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:49:16 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/331b3634-6db3-4444-ab39-8fa3073bd7b2 {"id":"331b3634-6db3-4444-ab39-8fa3073bd7b2","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996153} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbc393747 331b3634-6db3-4444-ab39-8fa3073bd7b2 /tidb/cdc/default/default/upstream/7365852314193066986 {"id":7365852314193066986,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/331b3634-6db3-4444-ab39-8fa3073bd7b2 {"id":"331b3634-6db3-4444-ab39-8fa3073bd7b2","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996153} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbc393747 331b3634-6db3-4444-ab39-8fa3073bd7b2 /tidb/cdc/default/default/upstream/7365852314193066986 {"id":7365852314193066986,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/331b3634-6db3-4444-ab39-8fa3073bd7b2 {"id":"331b3634-6db3-4444-ab39-8fa3073bd7b2","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996153} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbc393747 331b3634-6db3-4444-ab39-8fa3073bd7b2 /tidb/cdc/default/default/upstream/7365852314193066986 {"id":7365852314193066986,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/c73e6550-2b13-4366-a73e-72990490cfa1 + [[ /tidb/cdc/default/__cdc_meta__/capture/c73e6550-2b13-4366-a73e-72990490cfa1 =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/c73e6550-2b13-4366-a73e-72990490cfa1' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/c73e6550-2b13-4366-a73e-72990490cfa1 + echo 'check failed' check failed + exit 1 run task failed 3-th time, retry later table ddl_manager.finish_mark not exists for 14-th check, retry later [Mon May 6 19:49:17 CST 2024] <<<<<< START Pulsar consumer in changefeed_reconstruct case >>>>>> ***************** properties ***************** "threadcount"="4" "requestdistribution"="uniform" "mysql.user"="root" "readallfields"="true" "scanproportion"="0" "recordcount"="50" "mysql.db"="changefeed_reconstruct" "operationcount"="0" "mysql.port"="4000" "readproportion"="0" "workload"="core" "mysql.host"="127.0.0.1" "dotransactions"="false" "updateproportion"="0" "insertproportion"="0" ********************************************** Run finished, takes 23.117696ms INSERT - Takes(s): 0.0, Count: 47, OPS: 3257.7, Avg(us): 1696, Min(us): 993, Max(us): 8623, 95th(us): 9000, 99th(us): 9000 table changefeed_reconstruct.usertable not exists for 1-th check, retry later public/default ++ date + echo '[Mon May 6 19:49:17 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:49:17 CST 2024] <<<<<< pulsar is ready >>>>>> [Mon May 6 19:49:17 CST 2024] <<<<<< START cdc server in changefeed_pause_resume case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_pause_resume.69016903.out server --log-file /tmp/tidb_cdc_test/changefeed_pause_resume/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_pause_resume/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/open_protocol_handle_key_only/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:49:17 CST 2024] <<<<<< run test case open_protocol_handle_key_only success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:49:18 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4d9911c0-fbf2-4553-8e37-b44c52edda84 {"id":"4d9911c0-fbf2-4553-8e37-b44c52edda84","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996155} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbc3a104b 4d9911c0-fbf2-4553-8e37-b44c52edda84 /tidb/cdc/default/default/upstream/7365852313343450083 {"id":7365852313343450083,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4d9911c0-fbf2-4553-8e37-b44c52edda84 {"id":"4d9911c0-fbf2-4553-8e37-b44c52edda84","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996155} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbc3a104b 4d9911c0-fbf2-4553-8e37-b44c52edda84 /tidb/cdc/default/default/upstream/7365852313343450083 {"id":7365852313343450083,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4d9911c0-fbf2-4553-8e37-b44c52edda84 {"id":"4d9911c0-fbf2-4553-8e37-b44c52edda84","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996155} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbc3a104b 4d9911c0-fbf2-4553-8e37-b44c52edda84 /tidb/cdc/default/default/upstream/7365852313343450083 {"id":7365852313343450083,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table ddl_manager.finish_mark not exists for 15-th check, retry later [Mon May 6 19:49:19 CST 2024] <<<<<< START Pulsar consumer in processor_stop_delay case >>>>>> table processor_stop_delay.t not exists for 1-th check, retry later table changefeed_reconstruct.usertable exists check diff failed 1-th time, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc) 3667975680 bytes in 20.12 secs (182343755 bytes/sec) [Pipeline] { [Pipeline] cache + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:49:20 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/afdd5318-57b4-4357-b44f-80bbc739e523 {"id":"afdd5318-57b4-4357-b44f-80bbc739e523","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996158} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbc4dad45 afdd5318-57b4-4357-b44f-80bbc739e523 /tidb/cdc/default/default/upstream/7365852330056153277 {"id":7365852330056153277,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/afdd5318-57b4-4357-b44f-80bbc739e523 {"id":"afdd5318-57b4-4357-b44f-80bbc739e523","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996158} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbc4dad45 afdd5318-57b4-4357-b44f-80bbc739e523 /tidb/cdc/default/default/upstream/7365852330056153277 {"id":7365852330056153277,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/afdd5318-57b4-4357-b44f-80bbc739e523 {"id":"afdd5318-57b4-4357-b44f-80bbc739e523","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996158} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbc4dad45 afdd5318-57b4-4357-b44f-80bbc739e523 /tidb/cdc/default/default/upstream/7365852330056153277 {"id":7365852330056153277,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/canal_json_claim_check/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:49:20 CST 2024] <<<<<< run test case canal_json_claim_check success! >>>>>> [Mon May 6 19:49:21 CST 2024] <<<<<< START Pulsar consumer in changefeed_pause_resume case >>>>>> table ddl_manager.finish_mark not exists for 16-th check, retry later check diff successfully table processor_stop_delay.t exists check diff failed 1-th time, retry later wait process 6471 exit for 1-th time... wait process 6471 exit for 2-th time... wait process 6471 exit for 3-th time... table ddl_manager.finish_mark not exists for 17-th check, retry later check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info= + [[ '' =~ capture ]] + echo 'check pass' check pass + exit 0 run task successfully check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Mon May 6 19:49:22 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/NewChangefeedRetryError=return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.68026804.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (6471) - No such process wait process 6471 exit for 4-th time... process 6471 already exit check_no_capture http://127.0.0.1:2379 parse error: Invalid numeric literal at line 1, column 6 check diff successfully run task successfully [Mon May 6 19:49:23 CST 2024] <<<<<< START cdc server in changefeed_reconstruct case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_reconstruct.68196821.out server --log-file /tmp/tidb_cdc_test/changefeed_reconstruct/cdcserver2.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_reconstruct/cdc_dataserver2 --cluster-id default --addr 127.0.0.1:8300 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table changefeed_pause_resume.t1 exists table changefeed_pause_resume.t2 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/open_protocol_claim_check/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:49:24 CST 2024] <<<<<< run test case open_protocol_claim_check success! >>>>>> table ddl_manager.finish_mark not exists for 18-th check, retry later check diff failed 1-th time, retry later table changefeed_pause_resume.t2 exists table changefeed_pause_resume.t3 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:49:26 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2a56efb1-9ec3-4634-94fa-91f9d9cfa2dd {"id":"2a56efb1-9ec3-4634-94fa-91f9d9cfa2dd","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996163} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbbf52367 2a56efb1-9ec3-4634-94fa-91f9d9cfa2dd /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7365852232033146232,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"pulsar://127.0.0.1:6650/ticdc-sink-retry-test-4702?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:49:02.802513524+08:00","start-ts":449575945484369921,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"pending","error":{"time":"2024-05-06T19:49:23.544989249+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575954292146178} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":449575950844952580,"min-table-barrier-ts":449575950844952580,"admin-job-type":1} /tidb/cdc/default/default/upstream/7365852232033146232 {"id":7365852232033146232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2a56efb1-9ec3-4634-94fa-91f9d9cfa2dd {"id":"2a56efb1-9ec3-4634-94fa-91f9d9cfa2dd","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996163} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbbf52367 2a56efb1-9ec3-4634-94fa-91f9d9cfa2dd /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7365852232033146232,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"pulsar://127.0.0.1:6650/ticdc-sink-retry-test-4702?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:49:02.802513524+08:00","start-ts":449575945484369921,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"pending","error":{"time":"2024-05-06T19:49:23.544989249+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575954292146178} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":449575950844952580,"min-table-barrier-ts":449575950844952580,"admin-job-type":1} /tidb/cdc/default/default/upstream/7365852232033146232 {"id":7365852232033146232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2a56efb1-9ec3-4634-94fa-91f9d9cfa2dd {"id":"2a56efb1-9ec3-4634-94fa-91f9d9cfa2dd","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996163} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbbf52367 2a56efb1-9ec3-4634-94fa-91f9d9cfa2dd /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7365852232033146232,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"pulsar://127.0.0.1:6650/ticdc-sink-retry-test-4702?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:49:02.802513524+08:00","start-ts":449575945484369921,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"pending","error":{"time":"2024-05-06T19:49:23.544989249+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575954292146178} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":449575950844952580,"min-table-barrier-ts":449575950844952580,"admin-job-type":1} /tidb/cdc/default/default/upstream/7365852232033146232 {"id":7365852232033146232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-error warning failpoint injected retriable error + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error + expected_state=warning + error_msg=failpoint + tls_dir=error + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error -s + info='{ "upstream_id": 7365852232033146232, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 449575950844952580, "checkpoint_time": "2024-05-06 19:49:10.379", "error": { "time": "2024-05-06T19:49:23.544989249+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } }' + echo '{ "upstream_id": 7365852232033146232, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 449575950844952580, "checkpoint_time": "2024-05-06 19:49:10.379", "error": { "time": "2024-05-06T19:49:23.544989249+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } }' { "upstream_id": 7365852232033146232, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 449575950844952580, "checkpoint_time": "2024-05-06 19:49:10.379", "error": { "time": "2024-05-06T19:49:23.544989249+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } } ++ echo '{' '"upstream_id":' 7365852232033146232, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"warning",' '"checkpoint_tso":' 449575950844952580, '"checkpoint_time":' '"2024-05-06' '19:49:10.379",' '"error":' '{' '"time":' '"2024-05-06T19:49:23.544989249+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrOwnerUnknown",' '"message":' '"failpoint' injected retriable 'error"' '}' '}' ++ jq -r .state + state=warning + [[ ! warning == \w\a\r\n\i\n\g ]] ++ echo '{' '"upstream_id":' 7365852232033146232, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"warning",' '"checkpoint_tso":' 449575950844952580, '"checkpoint_time":' '"2024-05-06' '19:49:10.379",' '"error":' '{' '"time":' '"2024-05-06T19:49:23.544989249+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrOwnerUnknown",' '"message":' '"failpoint' injected retriable 'error"' '}' '}' ++ jq -r .error.message + message='failpoint injected retriable error' + [[ ! failpoint injected retriable error =~ failpoint ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6904.out cli changefeed remove -c changefeed-error + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:49:26 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/1104d40e-56c2-431f-b3d9-f58213eafef5 {UpstreamID:7365852314193066986 Namespace:default ID:1104d40e-56c2-431f-b3d9-f58213eafef5 SinkURI:pulsar://127.0.0.1:6650/ticdc-changefeed-reconstruct-16352?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:49:17.116921617 +0800 CST StartTs:449575952451633154 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0024f8510 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575952608919554} {CheckpointTs:449575955020644356 MinTableBarrierTs:449575955020644356 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449575955020644358, checkpointTs: 449575955020644356, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/67eaa1d6-ad8b-4104-b0d0-6fcf7d265554 {"id":"67eaa1d6-ad8b-4104-b0d0-6fcf7d265554","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996164} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbc3937eb 67eaa1d6-ad8b-4104-b0d0-6fcf7d265554 /tidb/cdc/default/default/changefeed/info/1104d40e-56c2-431f-b3d9-f58213eafef5 {"upstream-id":7365852314193066986,"namespace":"default","changefeed-id":"1104d40e-56c2-431f-b3d9-f58213eafef5","sink-uri":"pulsar://127.0.0.1:6650/ticdc-changefeed-reconstruct-16352?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:49:17.116921617+08:00","start-ts":449575952451633154,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575952608919554} /tidb/cdc/default/default/changefeed/status/1104d40e-56c2-431f-b3d9-f58213eafef5 {"checkpoint-ts":449575955020644356,"min-table-barrier-ts":449575955020644356,"admin-job-type":0} /tidb/cdc/default/default/task/position/67eaa1d6-ad8b-4104-b0d0-6fcf7d265554/1104d40e-56c2-431f-b3d9-f58213eafef5 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852314193066986 {"id":7365852314193066986,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/1104d40e-56c2-431f-b3d9-f58213eafef5 {UpstreamID:7365852314193066986 Namespace:default ID:1104d40e-56c2-431f-b3d9-f58213eafef5 SinkURI:pulsar://127.0.0.1:6650/ticdc-changefeed-reconstruct-16352?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:49:17.116921617 +0800 CST StartTs:449575952451633154 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0024f8510 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575952608919554} {CheckpointTs:449575955020644356 MinTableBarrierTs:449575955020644356 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449575955020644358, checkpointTs: 449575955020644356, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/67eaa1d6-ad8b-4104-b0d0-6fcf7d265554 {"id":"67eaa1d6-ad8b-4104-b0d0-6fcf7d265554","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996164} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbc3937eb 67eaa1d6-ad8b-4104-b0d0-6fcf7d265554 /tidb/cdc/default/default/changefeed/info/1104d40e-56c2-431f-b3d9-f58213eafef5 {"upstream-id":7365852314193066986,"namespace":"default","changefeed-id":"1104d40e-56c2-431f-b3d9-f58213eafef5","sink-uri":"pulsar://127.0.0.1:6650/ticdc-changefeed-reconstruct-16352?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:49:17.116921617+08:00","start-ts":449575952451633154,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575952608919554} /tidb/cdc/default/default/changefeed/status/1104d40e-56c2-431f-b3d9-f58213eafef5 {"checkpoint-ts":449575955020644356,"min-table-barrier-ts":449575955020644356,"admin-job-type":0} /tidb/cdc/default/default/task/position/67eaa1d6-ad8b-4104-b0d0-6fcf7d265554/1104d40e-56c2-431f-b3d9-f58213eafef5 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852314193066986 {"id":7365852314193066986,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/1104d40e-56c2-431f-b3d9-f58213eafef5 {UpstreamID:7365852314193066986 Namespace:default ID:1104d40e-56c2-431f-b3d9-f58213eafef5 SinkURI:pulsar://127.0.0.1:6650/ticdc-changefeed-reconstruct-16352?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:49:17.116921617 +0800 CST StartTs:449575952451633154 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0024f8510 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575952608919554} {CheckpointTs:449575955020644356 MinTableBarrierTs:449575955020644356 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449575955020644358, checkpointTs: 449575955020644356, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/67eaa1d6-ad8b-4104-b0d0-6fcf7d265554 {"id":"67eaa1d6-ad8b-4104-b0d0-6fcf7d265554","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996164} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbc3937eb 67eaa1d6-ad8b-4104-b0d0-6fcf7d265554 /tidb/cdc/default/default/changefeed/info/1104d40e-56c2-431f-b3d9-f58213eafef5 {"upstream-id":7365852314193066986,"namespace":"default","changefeed-id":"1104d40e-56c2-431f-b3d9-f58213eafef5","sink-uri":"pulsar://127.0.0.1:6650/ticdc-changefeed-reconstruct-16352?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:49:17.116921617+08:00","start-ts":449575952451633154,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575952608919554} /tidb/cdc/default/default/changefeed/status/1104d40e-56c2-431f-b3d9-f58213eafef5 {"checkpoint-ts":449575955020644356,"min-table-barrier-ts":449575955020644356,"admin-job-type":0} /tidb/cdc/default/default/task/position/67eaa1d6-ad8b-4104-b0d0-6fcf7d265554/1104d40e-56c2-431f-b3d9-f58213eafef5 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852314193066986 {"id":7365852314193066986,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list --pd=http://127.0.0.1:2379 2>&1 | grep id table ddl_manager.finish_mark not exists for 19-th check, retry later Changefeed remove successfully. ID: changefeed-error CheckpointTs: 449575950844952580 SinkURI: pulsar://127.0.0.1:6650/ticdc-sink-retry-test-4702?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... check diff failed 2-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/canal_json_storage_basic/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:49:27 CST 2024] <<<<<< run test case canal_json_storage_basic success! >>>>>> table changefeed_pause_resume.t3 exists "id": "67eaa1d6-ad8b-4104-b0d0-6fcf7d265554", "cluster-id": "default" run task successfully + set +x check_no_changefeed 127.0.0.1:2379 parse error: Invalid numeric literal at line 1, column 6 run task successfully wait process cdc.test exit for 1-th time... check diff failed 3-th time, retry later \033[0;36m<<< Run all test success >>>\033[0m table ddl_manager.finish_mark not exists for 20-th check, retry later wait process cdc.test exit for 2-th time... capture_id: 67eaa1d6-ad8b-4104-b0d0-6fcf7d265554 check_processor_table_count http://127.0.0.1:2379 1104d40e-56c2-431f-b3d9-f58213eafef5 67eaa1d6-ad8b-4104-b0d0-6fcf7d265554 1 run task successfully cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Mon May 6 19:49:30 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/InjectChangefeedDDLError=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.70127014.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check_processor_table_count http://127.0.0.1:2379 1104d40e-56c2-431f-b3d9-f58213eafef5 67eaa1d6-ad8b-4104-b0d0-6fcf7d265554 0 table count 1 does equal to expected count 0 run task failed 1-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/canal_json_storage_partition_table/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:49:30 CST 2024] <<<<<< run test case canal_json_storage_partition_table success! >>>>>> table ddl_manager.finish_mark not exists for 21-th check, retry later check diff failed 1-th time, retry later check diff failed 4-th time, retry later check_processor_table_count http://127.0.0.1:2379 1104d40e-56c2-431f-b3d9-f58213eafef5 67eaa1d6-ad8b-4104-b0d0-6fcf7d265554 0 run task successfully ***************** properties ***************** "updateproportion"="0" "insertproportion"="0" "threadcount"="4" "requestdistribution"="uniform" "mysql.user"="root" "readproportion"="0" "recordcount"="50" "scanproportion"="0" "mysql.host"="127.0.0.1" "mysql.db"="changefeed_reconstruct" "operationcount"="0" "dotransactions"="false" "workload"="core" "readallfields"="true" "mysql.port"="4000" ********************************************** Run finished, takes 15.589785ms INSERT - Takes(s): 0.0, Count: 48, OPS: 4015.7, Avg(us): 1237, Min(us): 834, Max(us): 3779, 95th(us): 4000, 99th(us): 4000 table changefeed_reconstruct.usertable not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 22-th check, retry later valid ~~~ running cdc Failed to start cdc, the usage tips should be printed 1st test case cdc_server_tips success! try an INVALID cdc server command [Mon May 6 19:49:31 CST 2024] <<<<<< START cdc server in cdc_server_tips case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ true != \n\o ]] + set +x + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.60496051.out server --log-file /tmp/tidb_cdc_test/cdc_server_tips/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cdc_server_tips/cdc_data --cluster-id default --pd None + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:49:33 GMT < Content-Length: 896 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/73809f3d-033d-4a7d-a9fb-ccaa6e425be6 {"id":"73809f3d-033d-4a7d-a9fb-ccaa6e425be6","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996170} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbbf523c0 73809f3d-033d-4a7d-a9fb-ccaa6e425be6 /tidb/cdc/default/default/upstream/7365852232033146232 {"id":7365852232033146232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/73809f3d-033d-4a7d-a9fb-ccaa6e425be6 {"id":"73809f3d-033d-4a7d-a9fb-ccaa6e425be6","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996170} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbbf523c0 73809f3d-033d-4a7d-a9fb-ccaa6e425be6 /tidb/cdc/default/default/upstream/7365852232033146232 {"id":7365852232033146232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/73809f3d-033d-4a7d-a9fb-ccaa6e425be6 {"id":"73809f3d-033d-4a7d-a9fb-ccaa6e425be6","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996170} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbbf523c0 73809f3d-033d-4a7d-a9fb-ccaa6e425be6 /tidb/cdc/default/default/upstream/7365852232033146232 {"id":7365852232033146232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.7068.out cli changefeed create --start-ts=449575945484369921 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-sink-retry-test-4702?protocol=canal-json&enable-tidb-extension=true' -c changefeed-error-1 Create changefeed successfully! ID: changefeed-error-1 Info: {"upstream_id":7365852232033146232,"namespace":"default","id":"changefeed-error-1","sink_uri":"pulsar://127.0.0.1:6650/ticdc-sink-retry-test-4702?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:49:33.787696007+08:00","start_ts":449575945484369921,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575945484369921,"checkpoint_ts":449575945484369921,"checkpoint_time":"2024-05-06 19:48:49.930"} PASS check diff failed 2-th time, retry later check diff failed 5-th time, retry later coverage: 2.6% of statements in github.com/pingcap/tiflow/... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/multi_tables_ddl/run.sh using Sink-Type: pulsar... <<================= * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:24927; Connection refused * Closing connection 0 You are running an older version of MinIO released 3 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend table ddl_manager.finish_mark not exists for 23-th check, retry later + set +x Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide table changefeed_reconstruct.usertable exists check diff failed 1-th time, retry later check diff failed 6-th time, retry later check_changefeed_status 127.0.0.1:8300 changefeed-error-1 warning last_warning ErrExecDDLFailed + endpoint=127.0.0.1:8300 + changefeed_id=changefeed-error-1 + expected_state=warning + field=last_warning + error_pattern=ErrExecDDLFailed ++ curl 127.0.0.1:8300/api/v2/changefeeds/changefeed-error-1/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 253 100 253 0 0 1863 0 --:--:-- --:--:-- --:--:-- 1874 + info='{"state":"warning","resolved_ts":449575945877323783,"checkpoint_ts":449575945877323783,"last_warning":{"time":"2024-05-06T19:49:35.659801931+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}}' + echo '{"state":"warning","resolved_ts":449575945877323783,"checkpoint_ts":449575945877323783,"last_warning":{"time":"2024-05-06T19:49:35.659801931+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}}' {"state":"warning","resolved_ts":449575945877323783,"checkpoint_ts":449575945877323783,"last_warning":{"time":"2024-05-06T19:49:35.659801931+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}} ++ jq -r .state ++ echo '{"state":"warning","resolved_ts":449575945877323783,"checkpoint_ts":449575945877323783,"last_warning":{"time":"2024-05-06T19:49:35.659801931+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec' DDL 'failed"}}' + state=warning + [[ ! warning == \w\a\r\n\i\n\g ]] + [[ -z last_warning ]] ++ echo '{"state":"warning","resolved_ts":449575945877323783,"checkpoint_ts":449575945877323783,"last_warning":{"time":"2024-05-06T19:49:35.659801931+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec' DDL 'failed"}}' ++ jq -r .last_warning.message + error_msg='[CDC:ErrExecDDLFailed]exec DDL failed' + [[ ! [CDC:ErrExecDDLFailed]exec DDL failed =~ ErrExecDDLFailed ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.7121.out cli changefeed remove -c changefeed-error-1 * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:24927 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 17CCE3EF847207A1 < X-Xss-Protection: 1; mode=block < Date: Mon, 06 May 2024 11:49:35 GMT < { [data not shown] * Connection #0 to host 127.0.0.1 left intact Bucket 's3://logbucket/' created Exiting on signal: INTERRUPT check diff failed 3-th time, retry later Changefeed remove successfully. ID: changefeed-error-1 CheckpointTs: 449575945877323783 SinkURI: pulsar://127.0.0.1:6650/ticdc-sink-retry-test-4702?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 24-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... + set +x check diff failed 4-th time, retry later wait process cdc.test exit for 2-th time... check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 1-th time... wait process cdc.test exit for 3-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 2-th time... \033[0;36m<<< Run all test success >>>\033[0m cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:49:39 CST 2024] <<<<<< run test case changefeed_reconstruct success! >>>>>> wait process cdc.test exit for 3-th time... table ddl_manager.finish_mark not exists for 25-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Mon May 6 19:49:39 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/txnutil/gc/InjectActualGCSafePoint=return(9223372036854775807)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.71887190.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:49:40 CST 2024] <<<<<< run test case processor_stop_delay success! >>>>>> check diff failed 5-th time, retry later table ddl_manager.finish_mark not exists for 26-th check, retry later check diff successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:49:42 GMT < Content-Length: 896 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1f42769c-ff89-43cd-a606-839cad2e153e {"id":"1f42769c-ff89-43cd-a606-839cad2e153e","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996180} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbbf52436 1f42769c-ff89-43cd-a606-839cad2e153e /tidb/cdc/default/default/upstream/7365852232033146232 {"id":7365852232033146232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1f42769c-ff89-43cd-a606-839cad2e153e {"id":"1f42769c-ff89-43cd-a606-839cad2e153e","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996180} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbbf52436 1f42769c-ff89-43cd-a606-839cad2e153e /tidb/cdc/default/default/upstream/7365852232033146232 {"id":7365852232033146232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1f42769c-ff89-43cd-a606-839cad2e153e {"id":"1f42769c-ff89-43cd-a606-839cad2e153e","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996180} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbbf52436 1f42769c-ff89-43cd-a606-839cad2e153e /tidb/cdc/default/default/upstream/7365852232033146232 {"id":7365852232033146232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.7240.out cli changefeed create --start-ts=449575945484369921 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-sink-retry-test-4702?protocol=canal-json&enable-tidb-extension=true' -c changefeed-error-2 Create changefeed successfully! ID: changefeed-error-2 Info: {"upstream_id":7365852232033146232,"namespace":"default","id":"changefeed-error-2","sink_uri":"pulsar://127.0.0.1:6650/ticdc-sink-retry-test-4702?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:49:43.300807595+08:00","start_ts":449575945484369921,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575945484369921,"checkpoint_ts":449575945484369921,"checkpoint_time":"2024-05-06 19:48:49.930"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 27-th check, retry later + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-error-2 failed [CDC:ErrSnapshotLostByGC] + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error-2 + expected_state=failed + error_msg='[CDC:ErrSnapshotLostByGC]' + tls_dir='[CDC:ErrSnapshotLostByGC]' + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error-2 -s + info='{ "upstream_id": 7365852232033146232, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 449575945484369921, "checkpoint_time": "2024-05-06 19:48:49.930", "error": { "time": "2024-05-06T19:49:43.378922266+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449575945484369921 is earlier than or equal to GC safepoint at 9223372036854775807" } }' + echo '{ "upstream_id": 7365852232033146232, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 449575945484369921, "checkpoint_time": "2024-05-06 19:48:49.930", "error": { "time": "2024-05-06T19:49:43.378922266+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449575945484369921 is earlier than or equal to GC safepoint at 9223372036854775807" } }' { "upstream_id": 7365852232033146232, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 449575945484369921, "checkpoint_time": "2024-05-06 19:48:49.930", "error": { "time": "2024-05-06T19:49:43.378922266+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449575945484369921 is earlier than or equal to GC safepoint at 9223372036854775807" } } ++ echo '{' '"upstream_id":' 7365852232033146232, '"namespace":' '"default",' '"id":' '"changefeed-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 449575945484369921, '"checkpoint_time":' '"2024-05-06' '19:48:49.930",' '"error":' '{' '"time":' '"2024-05-06T19:49:43.378922266+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSnapshotLostByGC",' '"message":' '"[CDC:ErrSnapshotLostByGC]fail' to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449575945484369921 is earlier than or equal to GC safepoint at '9223372036854775807"' '}' '}' ++ jq -r .state + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ echo '{' '"upstream_id":' 7365852232033146232, '"namespace":' '"default",' '"id":' '"changefeed-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 449575945484369921, '"checkpoint_time":' '"2024-05-06' '19:48:49.930",' '"error":' '{' '"time":' '"2024-05-06T19:49:43.378922266+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSnapshotLostByGC",' '"message":' '"[CDC:ErrSnapshotLostByGC]fail' to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449575945484369921 is earlier than or equal to GC safepoint at '9223372036854775807"' '}' '}' ++ jq -r .error.message + message='[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449575945484369921 is earlier than or equal to GC safepoint at 9223372036854775807' + [[ ! [CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449575945484369921 is earlier than or equal to GC safepoint at 9223372036854775807 =~ \[CDC:ErrSnapshotLostByGC] ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.7326.out cli changefeed remove -c changefeed-error-2 Changefeed remove successfully. ID: changefeed-error-2 CheckpointTs: 449575945484369921 SinkURI: pulsar://127.0.0.1:6650/ticdc-sink-retry-test-4702?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 28-th check, retry later check diff failed 1-th time, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc) 3667975680 bytes in 22.50 secs (163010580 bytes/sec) [Pipeline] { [Pipeline] sh [Pipeline] sh + set +x + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G12 Run cases: many_pk_or_uk capture_session_done_during_task ddl_attributes PROW_JOB_ID=cd0f73e1-2b01-4d7c-b838-cad499e0f373 JENKINS_NODE_COOKIE=0bb99d63-2fb9-45a3-a92a-f04f6f83bbb1 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-txbrj HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test","buildid":"1787445299229233158","prowjobid":"cd0f73e1-2b01-4d7c-b838-cad499e0f373","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/usr/java/openjdk-17/bin:/usr/local/pulsar/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#72 TEST_GROUP=G12 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233158 GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-5flxm GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-49db5c2ac89dce8b54933296f8c1a6663a0148762a9e3dd665403ece56205519 NODE_LABELS=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-txbrj ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-5flxm GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-txbrj GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=72 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/many_pk_or_uk/run.sh using Sink-Type: pulsar... <<================= [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G14 Run cases: changefeed_finish force_replicate_table PROW_JOB_ID=cd0f73e1-2b01-4d7c-b838-cad499e0f373 JENKINS_NODE_COOKIE=fe4b3550-7a8d-4497-ac81-b95f1e87299a BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-96v7q HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test","buildid":"1787445299229233158","prowjobid":"cd0f73e1-2b01-4d7c-b838-cad499e0f373","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/usr/java/openjdk-17/bin:/usr/local/pulsar/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#72 TEST_GROUP=G14 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233158 GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-n4fbv GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-49db5c2ac89dce8b54933296f8c1a6663a0148762a9e3dd665403ece56205519 NODE_LABELS=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-n4fbv ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-96v7q GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-96v7q GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=72 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/changefeed_finish/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh wait process cdc.test exit for 1-th time... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G15 Run cases: new_ci_collation batch_add_table multi_rocks PROW_JOB_ID=cd0f73e1-2b01-4d7c-b838-cad499e0f373 JENKINS_NODE_COOKIE=221f69b4-0c04-4bed-8ccb-3ca6af950771 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-08gz3 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test","buildid":"1787445299229233158","prowjobid":"cd0f73e1-2b01-4d7c-b838-cad499e0f373","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/usr/java/openjdk-17/bin:/usr/local/pulsar/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#72 TEST_GROUP=G15 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233158 GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-r63qz GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-49db5c2ac89dce8b54933296f8c1a6663a0148762a9e3dd665403ece56205519 NODE_LABELS=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-08gz3 ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-r63qz GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-08gz3 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=72 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/new_ci_collation/run.sh using Sink-Type: pulsar... <<================= [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G13 Run cases: tiflash region_merge common_1 PROW_JOB_ID=cd0f73e1-2b01-4d7c-b838-cad499e0f373 JENKINS_NODE_COOKIE=6b236909-2a42-4563-8584-78b59b2d2708 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-qx8wr HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test","buildid":"1787445299229233158","prowjobid":"cd0f73e1-2b01-4d7c-b838-cad499e0f373","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/usr/java/openjdk-17/bin:/usr/local/pulsar/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#72 TEST_GROUP=G13 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233158 GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-wpwbl GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-49db5c2ac89dce8b54933296f8c1a6663a0148762a9e3dd665403ece56205519 NODE_LABELS=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-wpwbl ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-qx8wr GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-qx8wr GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=72 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/tiflash/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh wait process cdc.test exit for 2-th time... table ddl_manager.finish_mark not exists for 29-th check, retry later check diff failed 2-th time, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Mon May 6 19:49:48 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/redo/ChangefeedNewRedoManagerError=2*return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.73757377.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] // podTemplate [Pipeline] } + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G16 + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh pulsar G17 Run cases: clustered_index processor_resolved_ts_fallback PROW_JOB_ID=cd0f73e1-2b01-4d7c-b838-cad499e0f373 JENKINS_NODE_COOKIE=733e928f-b462-4eea-a0c1-1fa5ffc893a8 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-z5vmx HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test","buildid":"1787445299229233158","prowjobid":"cd0f73e1-2b01-4d7c-b838-cad499e0f373","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/usr/java/openjdk-17/bin:/usr/local/pulsar/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#72 TEST_GROUP=G17 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233158 GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-cb666 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-49db5c2ac89dce8b54933296f8c1a6663a0148762a9e3dd665403ece56205519 NODE_LABELS=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-cb666 ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-z5vmx GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-z5vmx GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=72 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/clustered_index/run.sh using Sink-Type: pulsar... <<================= [Mon May 6 19:49:49 CST 2024] <<<<<< skip test case clustered_index for pulsar! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/processor_resolved_ts_fallback/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... [Pipeline] // stage [Pipeline] } Run cases: owner_resign processor_etcd_worker_delay sink_hang PROW_JOB_ID=cd0f73e1-2b01-4d7c-b838-cad499e0f373 JENKINS_NODE_COOKIE=93512a5c-00cc-4957-b6c6-7c39ced9a5db BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-807c6 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test","buildid":"1787445299229233158","prowjobid":"cd0f73e1-2b01-4d7c-b838-cad499e0f373","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-8.1","base_sha":"38eb040812f817df410abe870952629df60fd01f","base_link":"https://github.com/pingcap/tiflow/commit/38eb040812f817df410abe870952629df60fd01f","pulls":[{"number":11026,"author":"ti-chi-bot","sha":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","title":"kafka(ticdc): Fix java example license issue (#11024)","link":"https://github.com/pingcap/tiflow/pull/11026","commit_link":"https://github.com/pingcap/tiflow/pull/11026/commits/e1a27adc69acc9d50db0c3fea2945bbd6d983e30","author_link":"https://github.com/ti-chi-bot"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=03312178c534dce949face80c69812d989e55009 PATH=/usr/java/openjdk-17/bin:/usr/local/pulsar/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#72 TEST_GROUP=G16 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1787445299229233158 GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_pulsar_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=a0a6bb5614d55afbab88f0d1ae459423175a5407 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/72/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-ddztn GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-49db5c2ac89dce8b54933296f8c1a6663a0148762a9e3dd665403ece56205519 NODE_LABELS=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-807c6 ap_tiflow_release-8_1_pull_cdc_integration_pulsar_test_72-ddztn GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-807c6 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-8.1/job/pull_cdc_integration_pulsar_test/display/redirect BUILD_NUMBER=72 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/owner_resign/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/multi_capture/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/partition_table/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table ddl_manager.finish_mark not exists for 30-th check, retry later check diff failed 3-th time, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/capture_session_done_during_task/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/changefeed_finish Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/batch_add_table/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/tiflash Starting Upstream PD... invalid ~~~ running cdc Failed to start cdc, the usage tips should be printed 2nd test case cdc_server_tips success! [Mon May 6 19:49:51 CST 2024] <<<<<< run all test cases cdc_server_tips success! >>>>>> Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:49:51 GMT < Content-Length: 896 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47a710e1-ae63-4aeb-9cd6-5c0544fa158f {"id":"47a710e1-ae63-4aeb-9cd6-5c0544fa158f","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996189} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 7 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbbf524aa 47a710e1-ae63-4aeb-9cd6-5c0544fa158f /tidb/cdc/default/default/upstream/7365852232033146232 {"id":7365852232033146232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47a710e1-ae63-4aeb-9cd6-5c0544fa158f {"id":"47a710e1-ae63-4aeb-9cd6-5c0544fa158f","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996189} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 7 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbbf524aa 47a710e1-ae63-4aeb-9cd6-5c0544fa158f /tidb/cdc/default/default/upstream/7365852232033146232 {"id":7365852232033146232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47a710e1-ae63-4aeb-9cd6-5c0544fa158f {"id":"47a710e1-ae63-4aeb-9cd6-5c0544fa158f","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996189} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 7 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbbf524aa 47a710e1-ae63-4aeb-9cd6-5c0544fa158f /tidb/cdc/default/default/upstream/7365852232033146232 {"id":7365852232033146232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.7430.out cli changefeed create --start-ts=0 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-sink-retry-test-4702?protocol=canal-json&enable-tidb-extension=true' -c changefeed-initialize-error check diff failed 4-th time, retry later Create changefeed successfully! ID: changefeed-initialize-error Info: {"upstream_id":7365852232033146232,"namespace":"default","id":"changefeed-initialize-error","sink_uri":"pulsar://127.0.0.1:6650/ticdc-sink-retry-test-4702?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:49:52.360454857+08:00","start_ts":449575961815941122,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575961815941122,"checkpoint_ts":449575961815941122,"checkpoint_time":"2024-05-06 19:49:52.230"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 31-th check, retry later Verifying downstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/partition_table Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s start tidb cluster in /tmp/tidb_cdc_test/capture_session_done_during_task Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... + info='{ "upstream_id": 7365852232033146232, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 449575961815941122, "checkpoint_time": "2024-05-06 19:49:52.230", "error": { "time": "2024-05-06T19:49:52.538596714+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' + echo '{ "upstream_id": 7365852232033146232, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 449575961815941122, "checkpoint_time": "2024-05-06 19:49:52.230", "error": { "time": "2024-05-06T19:49:52.538596714+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' { "upstream_id": 7365852232033146232, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 449575961815941122, "checkpoint_time": "2024-05-06 19:49:52.230", "error": { "time": "2024-05-06T19:49:52.538596714+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } } ++ echo '{' '"upstream_id":' 7365852232033146232, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"warning",' '"checkpoint_tso":' 449575961815941122, '"checkpoint_time":' '"2024-05-06' '19:49:52.230",' '"error":' '{' '"time":' '"2024-05-06T19:49:52.538596714+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .state + state=warning + [[ ! warning == \n\o\r\m\a\l ]] + echo 'changefeed state warning does not equal to normal' changefeed state warning does not equal to normal + exit 1 run task failed 1-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/batch_add_table Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... table ddl_manager.finish_mark not exists for 32-th check, retry later check diff failed 5-th time, retry later Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + info='{ "upstream_id": 7365852232033146232, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 449575962654801923, "checkpoint_time": "2024-05-06 19:49:55.430", "error": null }' + echo '{ "upstream_id": 7365852232033146232, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 449575962654801923, "checkpoint_time": "2024-05-06 19:49:55.430", "error": null }' { "upstream_id": 7365852232033146232, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 449575962654801923, "checkpoint_time": "2024-05-06 19:49:55.430", "error": null } ++ echo '{' '"upstream_id":' 7365852232033146232, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 449575962654801923, '"checkpoint_time":' '"2024-05-06' '19:49:55.430",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7365852232033146232, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 449575962654801923, '"checkpoint_time":' '"2024-05-06' '19:49:55.430",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.7557.out cli changefeed pause -c changefeed-initialize-error table ddl_manager.finish_mark not exists for 33-th check, retry later check diff successfully PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... start tidb cluster in /tmp/tidb_cdc_test/processor_resolved_ts_fallback Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/owner_resign Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/multi_capture Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error stopped changefeed new redo manager injected error + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=stopped + error_msg=changefeed + tls_dir=error + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + info='{ "upstream_id": 7365852232033146232, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 449575962654801923, "checkpoint_time": "2024-05-06 19:49:55.430", "error": { "time": "2024-05-06T19:49:52.538596714+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' + echo '{ "upstream_id": 7365852232033146232, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 449575962654801923, "checkpoint_time": "2024-05-06 19:49:55.430", "error": { "time": "2024-05-06T19:49:52.538596714+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' { "upstream_id": 7365852232033146232, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 449575962654801923, "checkpoint_time": "2024-05-06 19:49:55.430", "error": { "time": "2024-05-06T19:49:52.538596714+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } } ++ echo '{' '"upstream_id":' 7365852232033146232, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"stopped",' '"checkpoint_tso":' 449575962654801923, '"checkpoint_time":' '"2024-05-06' '19:49:55.430",' '"error":' '{' '"time":' '"2024-05-06T19:49:52.538596714+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7365852232033146232, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"stopped",' '"checkpoint_tso":' 449575962654801923, '"checkpoint_time":' '"2024-05-06' '19:49:55.430",' '"error":' '{' '"time":' '"2024-05-06T19:49:52.538596714+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .error.message + message='changefeed new redo manager injected error' + [[ ! changefeed new redo manager injected error =~ changefeed ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.7649.out cli changefeed resume -c changefeed-initialize-error table ddl_manager.finish_mark not exists for 34-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release PASS Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release coverage: 2.3% of statements in github.com/pingcap/tiflow/... Verifying downstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 35-th check, retry later + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + info='{ "upstream_id": 7365852232033146232, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 449575962654801923, "checkpoint_time": "2024-05-06 19:49:55.430", "error": null }' + echo '{ "upstream_id": 7365852232033146232, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 449575962654801923, "checkpoint_time": "2024-05-06 19:49:55.430", "error": null }' { "upstream_id": 7365852232033146232, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 449575962654801923, "checkpoint_time": "2024-05-06 19:49:55.430", "error": null } ++ echo '{' '"upstream_id":' 7365852232033146232, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 449575962654801923, '"checkpoint_time":' '"2024-05-06' '19:49:55.430",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7365852232033146232, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 449575962654801923, '"checkpoint_time":' '"2024-05-06' '19:49:55.430",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.7739.out cli changefeed remove -c changefeed-initialize-error check diff failed 1-th time, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Changefeed remove successfully. ID: changefeed-initialize-error CheckpointTs: 449575963965521924 SinkURI: pulsar://127.0.0.1:6650/ticdc-sink-retry-test-4702?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc already exists) [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc already exists) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // withEnv [Pipeline] } [Pipeline] // container ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later table ddl_manager.finish_mark not exists for 36-th check, retry later + set +x VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f52ebc0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-96v7q, pid:1224, start at 2024-05-06 19:50:02.436972901 +0800 CST m=+5.192283692 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:02.445 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:02.415 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:02.415 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f53738000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-qx8wr, pid:1227, start at 2024-05-06 19:50:02.973046929 +0800 CST m=+5.124898646 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:02.979 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:02.958 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:02.958 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f53738000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-qx8wr, pid:1227, start at 2024-05-06 19:50:02.973046929 +0800 CST m=+5.124898646 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:02.979 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:02.958 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:02.958 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f539840015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-qx8wr, pid:1298, start at 2024-05-06 19:50:03.141415902 +0800 CST m=+5.226371268 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:03.147 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:03.105 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:03.105 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/tiflash/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/tiflash/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // cache ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // cache check diff failed 3-th time, retry later [Pipeline] } [Pipeline] } [Pipeline] // stage [Pipeline] } wait process cdc.test exit for 3-th time... table ddl_manager.finish_mark not exists for 37-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:50:05 CST 2024] <<<<<< run test case changefeed_error success! >>>>>> [Pipeline] // withEnv ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] } [Pipeline] // dir [Pipeline] // dir [Pipeline] } [Pipeline] } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // node [Pipeline] } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f52ebc0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-96v7q, pid:1224, start at 2024-05-06 19:50:02.436972901 +0800 CST m=+5.192283692 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:02.445 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:02.415 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:02.415 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f530400015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-96v7q, pid:1307, start at 2024-05-06 19:50:02.548545473 +0800 CST m=+5.241205029 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:02.554 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:02.512 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:02.512 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_finish/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_finish/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Pipeline] // timeout VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f539440017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-k03d6, pid:10967, start at 2024-05-06 19:50:03.138700701 +0800 CST m=+5.300070581 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:03.146 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:03.139 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:03.139 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f539440017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-k03d6, pid:10967, start at 2024-05-06 19:50:03.138700701 +0800 CST m=+5.300070581 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:03.146 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:03.139 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:03.139 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f539ec0015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-k03d6, pid:11045, start at 2024-05-06 19:50:03.172502645 +0800 CST m=+5.281432581 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:03.180 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:03.181 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:03.181 +0800 All versions after safe point can be accessed. (DO NOT EDIT) [Pipeline] // timeout [Pipeline] } [Pipeline] } [Pipeline] // podTemplate [Pipeline] } Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/partition_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/partition_table/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tiflash.cli.2689.out cli tso query --pd=http://127.0.0.1:2379 [Pipeline] // stage [Pipeline] // stage VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f557800010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-08gz3, pid:1420, start at 2024-05-06 19:50:05.040712435 +0800 CST m=+5.028781173 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:05.046 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:05.024 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:05.024 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f557800010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-08gz3, pid:1420, start at 2024-05-06 19:50:05.040712435 +0800 CST m=+5.028781173 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:05.046 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:05.024 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:05.024 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f55aa4000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-08gz3, pid:1509, start at 2024-05-06 19:50:05.244593257 +0800 CST m=+5.175395831 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:05.250 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:05.225 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:05.225 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Pipeline] } [Pipeline] } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f558a00012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-txbrj, pid:1363, start at 2024-05-06 19:50:05.130566065 +0800 CST m=+5.145531991 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:05.137 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:05.096 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:05.096 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f558a00012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-txbrj, pid:1363, start at 2024-05-06 19:50:05.130566065 +0800 CST m=+5.145531991 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:05.137 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:05.096 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:05.096 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f5595c000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-txbrj, pid:1443, start at 2024-05-06 19:50:05.158392743 +0800 CST m=+5.116580171 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:05.165 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:05.143 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:05.143 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Pipeline] // withEnv [Pipeline] } [Pipeline] // container [Pipeline] // container [Pipeline] } [Pipeline] } table ddl_manager.finish_mark not exists for 38-th check, retry later [Pipeline] // stage [Pipeline] } [Pipeline] // withEnv [Pipeline] // withEnv [Pipeline] } [Pipeline] } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f563c80012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-z5vmx, pid:1233, start at 2024-05-06 19:50:05.838998582 +0800 CST m=+5.183559560 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:05.845 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:05.810 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:05.810 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // node check diff failed 4-th time, retry later [Pipeline] // node VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f56a88000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-807c6, pid:1333, start at 2024-05-06 19:50:06.256297893 +0800 CST m=+5.220126562 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:06.263 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:06.242 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:06.242 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] } [Pipeline] } [Pipeline] // podTemplate ++ date + echo '[Mon May 6 19:50:07 CST 2024] <<<<<< START pulsar cluster in changefeed_finish case >>>>>>' [Mon May 6 19:50:07 CST 2024] <<<<<< START pulsar cluster in changefeed_finish case >>>>>> + workdir=/tmp/tidb_cdc_test/changefeed_finish + cluster_type=normal + cd /tmp/tidb_cdc_test/changefeed_finish + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_finish/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/changefeed_finish/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/changefeed_finish/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_finish/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/changefeed_finish/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/changefeed_finish/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/changefeed_finish/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/changefeed_finish/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_finish/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/changefeed_finish/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/changefeed_finish/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/changefeed_finish/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:50:07 CST 2024] <<<<<< START pulsar cluster in normal mode in changefeed_finish case >>>>>>' [Mon May 6 19:50:07 CST 2024] <<<<<< START pulsar cluster in normal mode in changefeed_finish case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/changefeed_finish/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/changefeed_finish/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/changefeed_finish/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 [Pipeline] // podTemplate [Pipeline] } [Pipeline] } [Pipeline] // withEnv [Pipeline] // withEnv [Pipeline] } [Pipeline] } + set +x + tso='449575965558833153 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575965558833153 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:50:07 CST 2024] <<<<<< START cdc server in tiflash case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tiflash.27282730.out server --log-file /tmp/tidb_cdc_test/tiflash/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/tiflash/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] // stage [Pipeline] // stage [Pipeline] } [Pipeline] } ++ date + echo '[Mon May 6 19:50:08 CST 2024] <<<<<< START pulsar cluster in capture_session_done_during_task case >>>>>>' [Mon May 6 19:50:08 CST 2024] <<<<<< START pulsar cluster in capture_session_done_during_task case >>>>>> + workdir=/tmp/tidb_cdc_test/capture_session_done_during_task + cluster_type=normal + cd /tmp/tidb_cdc_test/capture_session_done_during_task + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/capture_session_done_during_task/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/capture_session_done_during_task/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/capture_session_done_during_task/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/capture_session_done_during_task/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/capture_session_done_during_task/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/capture_session_done_during_task/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/capture_session_done_during_task/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/capture_session_done_during_task/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/capture_session_done_during_task/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/capture_session_done_during_task/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/capture_session_done_during_task/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/capture_session_done_during_task/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:50:08 CST 2024] <<<<<< START pulsar cluster in normal mode in capture_session_done_during_task case >>>>>>' [Mon May 6 19:50:08 CST 2024] <<<<<< START pulsar cluster in normal mode in capture_session_done_during_task case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/capture_session_done_during_task/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/capture_session_done_during_task/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/capture_session_done_during_task/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.cli.12329.out cli tso query --pd=http://127.0.0.1:2379 table ddl_manager.finish_mark not exists for 39-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f57c80000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6, pid:8178, start at 2024-05-06 19:50:07.403727754 +0800 CST m=+5.385934340 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:07.409 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:07.392 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:07.392 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f57c80000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6, pid:8178, start at 2024-05-06 19:50:07.403727754 +0800 CST m=+5.385934340 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:07.409 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:07.392 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:07.392 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f57d2c0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6, pid:8247, start at 2024-05-06 19:50:07.479655278 +0800 CST m=+5.411675494 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:07.485 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:07.485 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:07.485 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_capture/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_capture/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f563c80012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-z5vmx, pid:1233, start at 2024-05-06 19:50:05.838998582 +0800 CST m=+5.183559560 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:05.845 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:05.810 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:05.810 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f563c00010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-z5vmx, pid:1314, start at 2024-05-06 19:50:05.826951222 +0800 CST m=+5.113719592 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:05.833 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:05.808 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:05.808 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 5-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f56a88000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-807c6, pid:1333, start at 2024-05-06 19:50:06.256297893 +0800 CST m=+5.220126562 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:06.263 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:06.242 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:06.242 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f56c080013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-807c6, pid:1419, start at 2024-05-06 19:50:06.36685583 +0800 CST m=+5.265044388 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:06.373 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:06.338 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:06.338 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/owner_resign/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/owner_resign/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 [Mon May 6 19:50:09 CST 2024] <<<<<< START cdc server in batch_add_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.27772779.out server --log-file /tmp/tidb_cdc_test/batch_add_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/batch_add_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/event_filter/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... + set +x + tso='449575966078402561 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575966078402561 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:50:10 CST 2024] <<<<<< START cdc server in partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.1236912371.out server --log-file /tmp/tidb_cdc_test/partition_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/partition_table/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 40-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/event_filter Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 check diff successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:50:11 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/18a9f524-ba67-4d6e-a72c-fbc024f2d495 {"id":"18a9f524-ba67-4d6e-a72c-fbc024f2d495","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996208} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd220a35 18a9f524-ba67-4d6e-a72c-fbc024f2d495 /tidb/cdc/default/default/upstream/7365852566800499864 {"id":7365852566800499864,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/18a9f524-ba67-4d6e-a72c-fbc024f2d495 {"id":"18a9f524-ba67-4d6e-a72c-fbc024f2d495","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996208} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd220a35 18a9f524-ba67-4d6e-a72c-fbc024f2d495 /tidb/cdc/default/default/upstream/7365852566800499864 {"id":7365852566800499864,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/18a9f524-ba67-4d6e-a72c-fbc024f2d495 {"id":"18a9f524-ba67-4d6e-a72c-fbc024f2d495","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996208} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd220a35 18a9f524-ba67-4d6e-a72c-fbc024f2d495 /tidb/cdc/default/default/upstream/7365852566800499864 {"id":7365852566800499864,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:50:11 CST 2024] <<<<<< START pulsar cluster in tiflash case >>>>>>' [Mon May 6 19:50:11 CST 2024] <<<<<< START pulsar cluster in tiflash case >>>>>> + workdir=/tmp/tidb_cdc_test/tiflash + cluster_type=mtls + cd /tmp/tidb_cdc_test/tiflash + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/tiflash/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/tiflash/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/tiflash/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/tiflash/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/tiflash/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/tiflash/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/tiflash/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/tiflash/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/tiflash/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/tiflash/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/tiflash/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/tiflash/pulsar_standalone.conf + pulsar_port=6650 + '[' mtls == mtls ']' + pulsar_port=6651 + gen_mtls_config + openssl genrsa -out ca.key.pem 2048 Generating RSA private key, 2048 bit long modulus .........+++ ...............................................+++ e is 65537 (0x10001) + openssl req -x509 -new -nodes -key ca.key.pem -subj /CN=CARoot -days 365 -out ca.cert.pem + openssl genrsa -out server.key.pem 2048 Generating RSA private key, 2048 bit long modulus ..................................................+++ ...........................................+++ e is 65537 (0x10001) + openssl pkcs8 -topk8 -inform PEM -outform PEM -in server.key.pem -out server.key-pk8.pem -nocrypt + echo '[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + openssl req -new -config server.conf -key server.key.pem -out server.csr.pem -sha256 + openssl x509 -req -in server.csr.pem -CA ca.cert.pem -CAkey ca.key.pem -CAcreateserial -out server.cert.pem -days 365 -extensions v3_ext -extfile server.conf -sha256 Signature ok subject=/CN=server Getting CA Private Key + openssl genrsa -out broker_client.key.pem 2048 Generating RSA private key, 2048 bit long modulus .......................+++ .............................................+++ e is 65537 (0x10001) + openssl pkcs8 -topk8 -inform PEM -outform PEM -in broker_client.key.pem -out broker_client.key-pk8.pem -nocrypt + openssl req -new -subj /CN=broker_client -key broker_client.key.pem -out broker_client.csr.pem -sha256 + openssl x509 -req -in broker_client.csr.pem -CA ca.cert.pem -CAkey ca.key.pem -CAcreateserial -out broker_client.cert.pem -days 365 -sha256 Signature ok subject=/CN=broker_client Getting CA Private Key + echo ' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/tiflash/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/tiflash/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/tiflash/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/tiflash/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/tiflash/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/tiflash/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' + echo ' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/tiflash/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/tiflash/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/tiflash/ca.cert.pem' ++ date + echo '[Mon May 6 19:50:11 CST 2024] <<<<<< START pulsar cluster in mtls mode in tiflash case >>>>>>' [Mon May 6 19:50:11 CST 2024] <<<<<< START pulsar cluster in mtls mode in tiflash case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6651 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/tiflash/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/tiflash/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/tiflash/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.cli.9640.out cli tso query --pd=http://127.0.0.1:2379 [Mon May 6 19:50:10 CST 2024] <<<<<< START cdc server in processor_resolved_ts_fallback case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkExecDMLError=return(true);github.com/pingcap/tiflow/cdc/sink/dmlsink/mq/dmlproducer/KafkaSinkAsyncSendError=return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.26982700.out server --log-file /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Mon May 6 19:50:11 CST 2024] <<<<<< START cdc server in owner_resign case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.27102714.out server --log-file /tmp/tidb_cdc_test/owner_resign/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/owner_resign/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:50:12 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/de863c2f-3c1a-41bd-bfd7-edf269f93031 {"id":"de863c2f-3c1a-41bd-bfd7-edf269f93031","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996209} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd2c736a de863c2f-3c1a-41bd-bfd7-edf269f93031 /tidb/cdc/default/default/upstream/7365852578626567578 {"id":7365852578626567578,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/de863c2f-3c1a-41bd-bfd7-edf269f93031 {"id":"de863c2f-3c1a-41bd-bfd7-edf269f93031","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996209} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd2c736a de863c2f-3c1a-41bd-bfd7-edf269f93031 /tidb/cdc/default/default/upstream/7365852578626567578 {"id":7365852578626567578,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/de863c2f-3c1a-41bd-bfd7-edf269f93031 {"id":"de863c2f-3c1a-41bd-bfd7-edf269f93031","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996209} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd2c736a de863c2f-3c1a-41bd-bfd7-edf269f93031 /tidb/cdc/default/default/upstream/7365852578626567578 {"id":7365852578626567578,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:50:12 CST 2024] <<<<<< START pulsar cluster in batch_add_table case >>>>>>' [Mon May 6 19:50:12 CST 2024] <<<<<< START pulsar cluster in batch_add_table case >>>>>> + workdir=/tmp/tidb_cdc_test/batch_add_table + cluster_type=normal + cd /tmp/tidb_cdc_test/batch_add_table + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/batch_add_table/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/batch_add_table/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/batch_add_table/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/batch_add_table/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/batch_add_table/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/batch_add_table/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/batch_add_table/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/batch_add_table/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/batch_add_table/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/batch_add_table/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/batch_add_table/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/batch_add_table/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:50:12 CST 2024] <<<<<< START pulsar cluster in normal mode in batch_add_table case >>>>>>' [Mon May 6 19:50:12 CST 2024] <<<<<< START pulsar cluster in normal mode in batch_add_table case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/batch_add_table/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/batch_add_table/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/batch_add_table/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 + set +x + tso='449575966826037249 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575966826037249 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x table ddl_manager.finish_mark not exists for 41-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:50:13 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c1e988d8-3542-4535-8bb1-a73cef6431a3 {"id":"c1e988d8-3542-4535-8bb1-a73cef6431a3","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996210} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd25dd39 c1e988d8-3542-4535-8bb1-a73cef6431a3 /tidb/cdc/default/default/upstream/7365852569381546856 {"id":7365852569381546856,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c1e988d8-3542-4535-8bb1-a73cef6431a3 {"id":"c1e988d8-3542-4535-8bb1-a73cef6431a3","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996210} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd25dd39 c1e988d8-3542-4535-8bb1-a73cef6431a3 /tidb/cdc/default/default/upstream/7365852569381546856 {"id":7365852569381546856,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c1e988d8-3542-4535-8bb1-a73cef6431a3 {"id":"c1e988d8-3542-4535-8bb1-a73cef6431a3","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996210} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd25dd39 c1e988d8-3542-4535-8bb1-a73cef6431a3 /tidb/cdc/default/default/upstream/7365852569381546856 {"id":7365852569381546856,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:50:13 CST 2024] <<<<<< START pulsar cluster in partition_table case >>>>>>' [Mon May 6 19:50:13 CST 2024] <<<<<< START pulsar cluster in partition_table case >>>>>> + workdir=/tmp/tidb_cdc_test/partition_table + cluster_type=normal + cd /tmp/tidb_cdc_test/partition_table + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/partition_table/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/partition_table/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/partition_table/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/partition_table/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/partition_table/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/partition_table/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/partition_table/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/partition_table/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/partition_table/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/partition_table/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/partition_table/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/partition_table/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:50:13 CST 2024] <<<<<< START pulsar cluster in normal mode in partition_table case >>>>>>' [Mon May 6 19:50:13 CST 2024] <<<<<< START pulsar cluster in normal mode in partition_table case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/partition_table/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/partition_table/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/partition_table/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 ***************** properties ***************** "mysql.db"="multi_capture_1" "readallfields"="true" "workload"="core" "threadcount"="2" "mysql.user"="root" "scanproportion"="0" "mysql.host"="127.0.0.1" "updateproportion"="0" "insertproportion"="0" "requestdistribution"="uniform" "recordcount"="10" "dotransactions"="false" "readproportion"="0" "operationcount"="0" "mysql.port"="4000" ********************************************** + nc -z localhost 6651 + i=2 + '[' 2 -gt 20 ']' + sleep 2 Tenant not found Reason: Tenant not found + i=1 + '[' 1 -gt 20 ']' + sleep 2 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:50:14 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/259d6627-52bc-491e-bab7-3c670bf8e816 {"id":"259d6627-52bc-491e-bab7-3c670bf8e816","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996211} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd32d631 259d6627-52bc-491e-bab7-3c670bf8e816 /tidb/cdc/default/default/upstream/7365852582901900235 {"id":7365852582901900235,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/259d6627-52bc-491e-bab7-3c670bf8e816 {"id":"259d6627-52bc-491e-bab7-3c670bf8e816","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996211} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd32d631 259d6627-52bc-491e-bab7-3c670bf8e816 /tidb/cdc/default/default/upstream/7365852582901900235 {"id":7365852582901900235,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/259d6627-52bc-491e-bab7-3c670bf8e816 {"id":"259d6627-52bc-491e-bab7-3c670bf8e816","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996211} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd32d631 259d6627-52bc-491e-bab7-3c670bf8e816 /tidb/cdc/default/default/upstream/7365852582901900235 {"id":7365852582901900235,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:50:14 CST 2024] <<<<<< START pulsar cluster in processor_resolved_ts_fallback case >>>>>>' [Mon May 6 19:50:14 CST 2024] <<<<<< START pulsar cluster in processor_resolved_ts_fallback case >>>>>> + workdir=/tmp/tidb_cdc_test/processor_resolved_ts_fallback + cluster_type=normal + cd /tmp/tidb_cdc_test/processor_resolved_ts_fallback + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/processor_resolved_ts_fallback/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/processor_resolved_ts_fallback/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/processor_resolved_ts_fallback/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/processor_resolved_ts_fallback/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/processor_resolved_ts_fallback/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/processor_resolved_ts_fallback/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/processor_resolved_ts_fallback/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/processor_resolved_ts_fallback/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/processor_resolved_ts_fallback/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/processor_resolved_ts_fallback/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/processor_resolved_ts_fallback/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:50:14 CST 2024] <<<<<< START pulsar cluster in normal mode in processor_resolved_ts_fallback case >>>>>>' [Mon May 6 19:50:14 CST 2024] <<<<<< START pulsar cluster in normal mode in processor_resolved_ts_fallback case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/processor_resolved_ts_fallback/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/processor_resolved_ts_fallback/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/processor_resolved_ts_fallback/pulsar-bookie Run finished, takes 11.260182ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1568.6, Avg(us): 2142, Min(us): 1258, Max(us): 4805, 95th(us): 5000, 99th(us): 5000 + i=1 + '[' 1 -gt 20 ']' + sleep 2 ++ date + echo '[Mon May 6 19:50:14 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:50:14 CST 2024] <<<<<< pulsar is ready >>>>>> + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table ddl_manager.finish_mark not exists for 42-th check, retry later ***************** properties ***************** "dotransactions"="false" "scanproportion"="0" "operationcount"="0" "threadcount"="2" "mysql.host"="127.0.0.1" "mysql.user"="root" "updateproportion"="0" "requestdistribution"="uniform" "insertproportion"="0" "readallfields"="true" "mysql.port"="4000" "mysql.db"="multi_capture_2" "workload"="core" "readproportion"="0" "recordcount"="10" ********************************************** + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:50:14 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7beb23e5-ad80-4c27-a174-147b06b1d9bb {"id":"7beb23e5-ad80-4c27-a174-147b06b1d9bb","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996211} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd345533 7beb23e5-ad80-4c27-a174-147b06b1d9bb /tidb/cdc/default/default/upstream/7365852581815843871 {"id":7365852581815843871,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7beb23e5-ad80-4c27-a174-147b06b1d9bb {"id":"7beb23e5-ad80-4c27-a174-147b06b1d9bb","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996211} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd345533 7beb23e5-ad80-4c27-a174-147b06b1d9bb /tidb/cdc/default/default/upstream/7365852581815843871 {"id":7365852581815843871,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7beb23e5-ad80-4c27-a174-147b06b1d9bb {"id":"7beb23e5-ad80-4c27-a174-147b06b1d9bb","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996211} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd345533 7beb23e5-ad80-4c27-a174-147b06b1d9bb /tidb/cdc/default/default/upstream/7365852581815843871 {"id":7365852581815843871,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc cli capture list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' 1 run task successfully ++ date + echo '[Mon May 6 19:50:14 CST 2024] <<<<<< START pulsar cluster in owner_resign case >>>>>>' [Mon May 6 19:50:14 CST 2024] <<<<<< START pulsar cluster in owner_resign case >>>>>> + workdir=/tmp/tidb_cdc_test/owner_resign + cluster_type=normal + cd /tmp/tidb_cdc_test/owner_resign + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/owner_resign/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/owner_resign/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/owner_resign/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/owner_resign/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/owner_resign/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/owner_resign/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/owner_resign/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/owner_resign/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/owner_resign/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/owner_resign/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/owner_resign/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/owner_resign/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:50:14 CST 2024] <<<<<< START pulsar cluster in normal mode in owner_resign case >>>>>>' [Mon May 6 19:50:14 CST 2024] <<<<<< START pulsar cluster in normal mode in owner_resign case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/owner_resign/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/owner_resign/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/owner_resign/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_session_done_during_task.cli.3207.out cli tso query --pd=http://127.0.0.1:2379 check diff failed 1-th time, retry later + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 Run finished, takes 393.058829ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1814.1, Avg(us): 78509, Min(us): 1026, Max(us): 387436, 95th(us): 388000, 99th(us): 388000 + nc -z localhost 6651 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + /usr/local/pulsar/bin/pulsar-admin namespaces list public + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 ***************** properties ***************** "dotransactions"="false" "workload"="core" "mysql.user"="root" "mysql.db"="multi_capture_3" "mysql.host"="127.0.0.1" "readproportion"="0" "threadcount"="2" "readallfields"="true" "operationcount"="0" "updateproportion"="0" "requestdistribution"="uniform" "mysql.port"="4000" "scanproportion"="0" "insertproportion"="0" "recordcount"="10" ********************************************** Run finished, takes 9.715849ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1829.2, Avg(us): 1838, Min(us): 1131, Max(us): 4157, 95th(us): 5000, 99th(us): 5000 + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 43-th check, retry later + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 check diff failed 2-th time, retry later + set +x + tso='449575967862554625 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575967862554625 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:50:16 CST 2024] <<<<<< START cdc server in capture_session_done_during_task case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorManagerHandleNewChangefeedDelay=sleep(2000)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_session_done_during_task.32513253.out server --log-file /tmp/tidb_cdc_test/capture_session_done_during_task/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/capture_session_done_during_task/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public public/default ***************** properties ***************** "threadcount"="2" "dotransactions"="false" "mysql.user"="root" "insertproportion"="0" "updateproportion"="0" "mysql.port"="4000" "readallfields"="true" "recordcount"="10" "scanproportion"="0" "requestdistribution"="uniform" "readproportion"="0" "operationcount"="0" "mysql.host"="127.0.0.1" "mysql.db"="multi_capture_4" "workload"="core" ********************************************** Run finished, takes 9.351628ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1829.4, Avg(us): 1736, Min(us): 982, Max(us): 3759, 95th(us): 4000, 99th(us): 4000 [Mon May 6 19:50:17 CST 2024] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.97709772.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ++ date + echo '[Mon May 6 19:50:18 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:50:18 CST 2024] <<<<<< pulsar is ready >>>>>> [Mon May 6 19:50:18 CST 2024] <<<<<< START cdc server in changefeed_finish case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_finish.31813183.out server --log-file /tmp/tidb_cdc_test/changefeed_finish/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_finish/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Tenant not found Reason: Tenant not found + i=1 + '[' 1 -gt 20 ']' + sleep 2 Tenant not found Reason: Tenant not found ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public table ddl_manager.finish_mark not exists for 44-th check, retry later + i=1 + '[' 1 -gt 20 ']' + sleep 2 + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public Tenant not found Reason: Tenant not found + i=1 + '[' 1 -gt 20 ']' + sleep 2 check diff failed 3-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/ddl_sequence/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... public/default + /usr/local/pulsar/bin/pulsar-admin namespaces list public + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:50:19 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/69989c57-c831-4b8b-91a5-120e3649f5fe {"id":"69989c57-c831-4b8b-91a5-120e3649f5fe","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996217} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd2a6266 69989c57-c831-4b8b-91a5-120e3649f5fe /tidb/cdc/default/default/upstream/7365852576963982018 {"id":7365852576963982018,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/69989c57-c831-4b8b-91a5-120e3649f5fe {"id":"69989c57-c831-4b8b-91a5-120e3649f5fe","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996217} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd2a6266 69989c57-c831-4b8b-91a5-120e3649f5fe /tidb/cdc/default/default/upstream/7365852576963982018 {"id":7365852576963982018,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/69989c57-c831-4b8b-91a5-120e3649f5fe {"id":"69989c57-c831-4b8b-91a5-120e3649f5fe","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996217} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd2a6266 69989c57-c831-4b8b-91a5-120e3649f5fe /tidb/cdc/default/default/upstream/7365852576963982018 {"id":7365852576963982018,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:50:20 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:50:20 CST 2024] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.cli.3156.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-processor-resolved-ts-fallback-test-18884?protocol=canal-json&enable-tidb-extension=true' --server=127.0.0.1:8301 table ddl_manager.finish_mark not exists for 45-th check, retry later Tenant not found Reason: Tenant not found + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:50:20 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d71f95a2-4784-49e9-849b-367b6dffcd6e {"id":"d71f95a2-4784-49e9-849b-367b6dffcd6e","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996217} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd3424a0 d71f95a2-4784-49e9-849b-367b6dffcd6e /tidb/cdc/default/default/upstream/7365852579471104142 {"id":7365852579471104142,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d71f95a2-4784-49e9-849b-367b6dffcd6e {"id":"d71f95a2-4784-49e9-849b-367b6dffcd6e","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996217} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd3424a0 d71f95a2-4784-49e9-849b-367b6dffcd6e /tidb/cdc/default/default/upstream/7365852579471104142 {"id":7365852579471104142,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d71f95a2-4784-49e9-849b-367b6dffcd6e {"id":"d71f95a2-4784-49e9-849b-367b6dffcd6e","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996217} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd3424a0 d71f95a2-4784-49e9-849b-367b6dffcd6e /tidb/cdc/default/default/upstream/7365852579471104142 {"id":7365852579471104142,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Mon May 6 19:50:20 CST 2024] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.98279829.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + i=1 + '[' 1 -gt 20 ']' + sleep 2 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:50:21 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47d4d2ba-bfd0-435f-80d4-d9f8e6c87a3c {"id":"47d4d2ba-bfd0-435f-80d4-d9f8e6c87a3c","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996218} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd1f8664 47d4d2ba-bfd0-435f-80d4-d9f8e6c87a3c /tidb/cdc/default/default/upstream/7365852558272560177 {"id":7365852558272560177,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47d4d2ba-bfd0-435f-80d4-d9f8e6c87a3c {"id":"47d4d2ba-bfd0-435f-80d4-d9f8e6c87a3c","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996218} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd1f8664 47d4d2ba-bfd0-435f-80d4-d9f8e6c87a3c /tidb/cdc/default/default/upstream/7365852558272560177 {"id":7365852558272560177,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47d4d2ba-bfd0-435f-80d4-d9f8e6c87a3c {"id":"47d4d2ba-bfd0-435f-80d4-d9f8e6c87a3c","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996218} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd1f8664 47d4d2ba-bfd0-435f-80d4-d9f8e6c87a3c /tidb/cdc/default/default/upstream/7365852558272560177 {"id":7365852558272560177,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check diff failed 4-th time, retry later Create changefeed successfully! ID: 896ad6db-673a-44f7-b199-571d7021e70e Info: {"upstream_id":7365852582901900235,"namespace":"default","id":"896ad6db-673a-44f7-b199-571d7021e70e","sink_uri":"pulsar://127.0.0.1:6650/ticdc-processor-resolved-ts-fallback-test-18884?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:50:21.314118636+08:00","start_ts":449575969295171587,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575969295171587,"checkpoint_ts":449575969295171587,"checkpoint_time":"2024-05-06 19:50:20.761"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... [Mon May 6 19:50:21 CST 2024] <<<<<< START Pulsar consumer in capture_session_done_during_task case >>>>>> lease 22318f4dbd2a6266 revoked + /usr/local/pulsar/bin/pulsar-admin namespaces list public [Mon May 6 19:50:21 CST 2024] <<<<<< START Pulsar consumer in changefeed_finish case >>>>>> check diff failed 1-th time, retry later public/default + set +x [Mon May 6 19:50:22 CST 2024] <<<<<< START Pulsar consumer in processor_resolved_ts_fallback case >>>>>> ++ date + echo '[Mon May 6 19:50:22 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:50:22 CST 2024] <<<<<< pulsar is ready >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table capture_session_done_during_task.t exists check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 46-th check, retry later + /usr/local/pulsar/bin/pulsar-admin namespaces list public cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' 1 run task successfully table processor_resolved_ts_fallback.t1 not exists for 1-th check, retry later public/default ++ date + echo '[Mon May 6 19:50:23 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:50:23 CST 2024] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.cli.3309.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-batch-add-table-test-32754?protocol=canal-json&enable-tidb-extension=true' check diff failed 5-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:50:23 GMT < Content-Length: 1297 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6d05d962-758d-4fbf-ab8b-a52cf2b5bcce {"id":"6d05d962-758d-4fbf-ab8b-a52cf2b5bcce","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996220} /tidb/cdc/default/__cdc_meta__/capture/d71f95a2-4784-49e9-849b-367b6dffcd6e {"id":"d71f95a2-4784-49e9-849b-367b6dffcd6e","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996217} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd3424a0 d71f95a2-4784-49e9-849b-367b6dffcd6e /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd3424d5 6d05d962-758d-4fbf-ab8b-a52cf2b5bcce /tidb/cdc/default/default/upstream/7365852579471104142 {"id":7365852579471104142,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6d05d962-758d-4fbf-ab8b-a52cf2b5bcce {"id":"6d05d962-758d-4fbf-ab8b-a52cf2b5bcce","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996220} /tidb/cdc/default/__cdc_meta__/capture/d71f95a2-4784-49e9-849b-367b6dffcd6e {"id":"d71f95a2-4784-49e9-849b-367b6dffcd6e","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996217} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd3424a0 d71f95a2-4784-49e9-849b-367b6dffcd6e /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd3424d5 6d05d962-758d-4fbf-ab8b-a52cf2b5bcce /tidb/cdc/default/default/upstream/7365852579471104142 {"id":7365852579471104142,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6d05d962-758d-4fbf-ab8b-a52cf2b5bcce {"id":"6d05d962-758d-4fbf-ab8b-a52cf2b5bcce","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996220} /tidb/cdc/default/__cdc_meta__/capture/d71f95a2-4784-49e9-849b-367b6dffcd6e {"id":"d71f95a2-4784-49e9-849b-367b6dffcd6e","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996217} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd3424a0 d71f95a2-4784-49e9-849b-367b6dffcd6e /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd3424d5 6d05d962-758d-4fbf-ab8b-a52cf2b5bcce /tidb/cdc/default/default/upstream/7365852579471104142 {"id":7365852579471104142,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Mon May 6 19:50:23 CST 2024] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.98859887.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc3.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data3 --cluster-id default --addr 127.0.0.1:8303 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8303 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8303; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Create changefeed successfully! ID: 93413df6-107d-4992-b8e1-e83913377ceb Info: {"upstream_id":7365852566800499864,"namespace":"default","id":"93413df6-107d-4992-b8e1-e83913377ceb","sink_uri":"pulsar+ssl://127.0.0.1:6651/ticdc-tiflash-test-11315?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:50:23.626337864+08:00","start_ts":449575965558833153,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"tls-trust-certs-file-path":"/tmp/tidb_cdc_test/tiflash/ca.cert.pem","connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30,"auth-tls-certificate-path":"/tmp/tidb_cdc_test/tiflash/broker_client.cert.pem","auth-tls-private-key-path":"/tmp/tidb_cdc_test/tiflash/broker_client.key-pk8.pem"},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575965558833153,"checkpoint_ts":449575965558833153,"checkpoint_time":"2024-05-06 19:50:06.508"} [Mon May 6 19:50:23 CST 2024] <<<<<< START Pulsar consumer in tiflash case >>>>>> public/default ++ date + echo '[Mon May 6 19:50:23 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:50:23 CST 2024] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.cli.12901.out cli changefeed create --start-ts=449575966078402561 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-partition-table-test-11289?protocol=canal-json&enable-tidb-extension=true' start tidb cluster in /tmp/tidb_cdc_test/ddl_sequence Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... Create changefeed successfully! ID: 00e8df27-3639-45b5-9e69-672aa0a14a29 Info: {"upstream_id":7365852578626567578,"namespace":"default","id":"00e8df27-3639-45b5-9e69-672aa0a14a29","sink_uri":"pulsar://127.0.0.1:6650/ticdc-batch-add-table-test-32754?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:50:24.111596962+08:00","start_ts":449575969993261060,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575969993261060,"checkpoint_ts":449575969993261060,"checkpoint_time":"2024-05-06 19:50:23.424"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... Create changefeed successfully! ID: d3ac7e15-a4df-41a0-9c10-4b0a70097b11 Info: {"upstream_id":7365852569381546856,"namespace":"default","id":"d3ac7e15-a4df-41a0-9c10-4b0a70097b11","sink_uri":"pulsar://127.0.0.1:6650/ticdc-partition-table-test-11289?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:50:24.816659487+08:00","start_ts":449575966078402561,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575966078402561,"checkpoint_ts":449575966078402561,"checkpoint_time":"2024-05-06 19:50:08.490"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f661000005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh, pid:7084, start at 2024-05-06 19:50:22.021043896 +0800 CST m=+5.407600159 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:22.028 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:22.016 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:22.016 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f661000005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh, pid:7084, start at 2024-05-06 19:50:22.021043896 +0800 CST m=+5.407600159 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:22.028 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:22.016 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:22.016 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f661b80014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh, pid:7161, start at 2024-05-06 19:50:22.102349218 +0800 CST m=+5.430454608 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:22.109 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:22.112 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:22.112 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/event_filter/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/event_filter/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 2-th time, retry later table ddl_manager.finish_mark not exists for 47-th check, retry later Verifying downstream PD is started... check diff failed 2-th time, retry later table cdc_tiflash_test.multi_data_type not exists for 1-th check, retry later public/default table processor_resolved_ts_fallback.t1 exists [Mon May 6 19:50:25 CST 2024] <<<<<< START cdc server in processor_resolved_ts_fallback case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.33013303.out server --log-file /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x [Mon May 6 19:50:25 CST 2024] <<<<<< START Pulsar consumer in batch_add_table case >>>>>> table batch_add_table.finish_mark not exists for 1-th check, retry later check diff successfully ++ date + echo '[Mon May 6 19:50:25 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:50:25 CST 2024] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.cli.3299.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-owner-resign-test-5698?protocol=canal-json&enable-tidb-extension=true' --server=127.0.0.1:8301 + set +x [Mon May 6 19:50:26 CST 2024] <<<<<< START Pulsar consumer in partition_table case >>>>>> table ddl_manager.finish_mark not exists for 48-th check, retry later Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8303 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8303 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8303 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:50:26 GMT < Content-Length: 1789 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/01f4c858-e4b1-449c-9234-3de85352b98f {"id":"01f4c858-e4b1-449c-9234-3de85352b98f","address":"127.0.0.1:8303","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996224} /tidb/cdc/default/__cdc_meta__/capture/6d05d962-758d-4fbf-ab8b-a52cf2b5bcce {"id":"6d05d962-758d-4fbf-ab8b-a52cf2b5bcce","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996220} /tidb/cdc/default/__cdc_meta__/capture/d71f95a2-4784-49e9-849b-367b6dffcd6e {"id":"d71f95a2-4784-49e9-849b-367b6dffcd6e","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996217} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd3424a0 d71f95a2-4784-49e9-849b-367b6dffcd6e /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd3424d5 6d05d962-758d-4fbf-ab8b-a52cf2b5bcce /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd3424f5 01f4c858-e4b1-449c-9234-3de85352b98f /tidb/cdc/default/default/upstream/7365852579471104142 {"id":7365852579471104142,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/01f4c858-e4b1-449c-9234-3de85352b98f {"id":"01f4c858-e4b1-449c-9234-3de85352b98f","address":"127.0.0.1:8303","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996224} /tidb/cdc/default/__cdc_meta__/capture/6d05d962-758d-4fbf-ab8b-a52cf2b5bcce {"id":"6d05d962-758d-4fbf-ab8b-a52cf2b5bcce","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996220} /tidb/cdc/default/__cdc_meta__/capture/d71f95a2-4784-49e9-849b-367b6dffcd6e {"id":"d71f95a2-4784-49e9-849b-367b6dffcd6e","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996217} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd3424a0 d71f95a2-4784-49e9-849b-367b6dffcd6e /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd3424d5 6d05d962-758d-4fbf-ab8b-a52cf2b5bcce /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd3424f5 01f4c858-e4b1-449c-9234-3de85352b98f /tidb/cdc/default/default/upstream/7365852579471104142 {"id":7365852579471104142,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/01f4c858-e4b1-449c-9234-3de85352b98f {"id":"01f4c858-e4b1-449c-9234-3de85352b98f","address":"127.0.0.1:8303","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996224} /tidb/cdc/default/__cdc_meta__/capture/6d05d962-758d-4fbf-ab8b-a52cf2b5bcce {"id":"6d05d962-758d-4fbf-ab8b-a52cf2b5bcce","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996220} /tidb/cdc/default/__cdc_meta__/capture/d71f95a2-4784-49e9-849b-367b6dffcd6e {"id":"d71f95a2-4784-49e9-849b-367b6dffcd6e","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996217} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd3424a0 d71f95a2-4784-49e9-849b-367b6dffcd6e /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd3424d5 6d05d962-758d-4fbf-ab8b-a52cf2b5bcce /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd3424f5 01f4c858-e4b1-449c-9234-3de85352b98f /tidb/cdc/default/default/upstream/7365852579471104142 {"id":7365852579471104142,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:50:26 CST 2024] <<<<<< START pulsar cluster in multi_capture case >>>>>>' [Mon May 6 19:50:26 CST 2024] <<<<<< START pulsar cluster in multi_capture case >>>>>> + workdir=/tmp/tidb_cdc_test/multi_capture + cluster_type=normal + cd /tmp/tidb_cdc_test/multi_capture + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/multi_capture/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/multi_capture/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/multi_capture/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/multi_capture/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/multi_capture/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/multi_capture/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/multi_capture/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/multi_capture/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/multi_capture/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/multi_capture/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/multi_capture/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/multi_capture/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:50:26 CST 2024] <<<<<< START pulsar cluster in normal mode in multi_capture case >>>>>>' [Mon May 6 19:50:26 CST 2024] <<<<<< START pulsar cluster in normal mode in multi_capture case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/multi_capture/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/multi_capture/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/multi_capture/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 Create changefeed successfully! ID: fcc4ed26-36b2-4416-bf3d-c61e2c6bd66b Info: {"upstream_id":7365852581815843871,"namespace":"default","id":"fcc4ed26-36b2-4416-bf3d-c61e2c6bd66b","sink_uri":"pulsar://127.0.0.1:6650/ticdc-owner-resign-test-5698?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:50:26.917353142+08:00","start_ts":449575970731982856,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575970731982856,"checkpoint_ts":449575970731982856,"checkpoint_time":"2024-05-06 19:50:26.242"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... [Mon May 6 19:50:26 CST 2024] <<<<<< START cdc server in event_filter case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.event_filter.85128514.out server --log-file /tmp/tidb_cdc_test/event_filter/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/event_filter/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff successfully check diff failed 1-th time, retry later table cdc_tiflash_test.multi_data_type exists check diff failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:50:28 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/896ad6db-673a-44f7-b199-571d7021e70e {UpstreamID:7365852582901900235 Namespace:default ID:896ad6db-673a-44f7-b199-571d7021e70e SinkURI:pulsar://127.0.0.1:6650/ticdc-processor-resolved-ts-fallback-test-18884?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:50:21.314118636 +0800 CST StartTs:449575969295171587 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002640ea0 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575969439350785} {CheckpointTs:449575971116810251 MinTableBarrierTs:449575971116810251 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/259d6627-52bc-491e-bab7-3c670bf8e816 {"id":"259d6627-52bc-491e-bab7-3c670bf8e816","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996211} /tidb/cdc/default/__cdc_meta__/capture/e2c60e32-b70c-4e3c-952e-076e8ac6e346 {"id":"e2c60e32-b70c-4e3c-952e-076e8ac6e346","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996225} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd32d631 259d6627-52bc-491e-bab7-3c670bf8e816 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd32d6d0 e2c60e32-b70c-4e3c-952e-076e8ac6e346 /tidb/cdc/default/default/changefeed/info/896ad6db-673a-44f7-b199-571d7021e70e {"upstream-id":7365852582901900235,"namespace":"default","changefeed-id":"896ad6db-673a-44f7-b199-571d7021e70e","sink-uri":"pulsar://127.0.0.1:6650/ticdc-processor-resolved-ts-fallback-test-18884?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:50:21.314118636+08:00","start-ts":449575969295171587,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575969439350785} /tidb/cdc/default/default/changefeed/status/896ad6db-673a-44f7-b199-571d7021e70e {"checkpoint-ts":449575971116810251,"min-table-barrier-ts":449575971116810251,"admin-job-type":0} /tidb/cdc/default/default/task/position/259d6627-52bc-491e-bab7-3c670bf8e816/896ad6db-673a-44f7-b199-571d7021e70e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/e2c60e32-b70c-4e3c-952e-076e8ac6e346/896ad6db-673a-44f7-b199-571d7021e70e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852582901900235 {"id":7365852582901900235,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/896ad6db-673a-44f7-b199-571d7021e70e {UpstreamID:7365852582901900235 Namespace:default ID:896ad6db-673a-44f7-b199-571d7021e70e SinkURI:pulsar://127.0.0.1:6650/ticdc-processor-resolved-ts-fallback-test-18884?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:50:21.314118636 +0800 CST StartTs:449575969295171587 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002640ea0 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575969439350785} {CheckpointTs:449575971116810251 MinTableBarrierTs:449575971116810251 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/259d6627-52bc-491e-bab7-3c670bf8e816 {"id":"259d6627-52bc-491e-bab7-3c670bf8e816","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996211} /tidb/cdc/default/__cdc_meta__/capture/e2c60e32-b70c-4e3c-952e-076e8ac6e346 {"id":"e2c60e32-b70c-4e3c-952e-076e8ac6e346","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996225} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd32d631 259d6627-52bc-491e-bab7-3c670bf8e816 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd32d6d0 e2c60e32-b70c-4e3c-952e-076e8ac6e346 /tidb/cdc/default/default/changefeed/info/896ad6db-673a-44f7-b199-571d7021e70e {"upstream-id":7365852582901900235,"namespace":"default","changefeed-id":"896ad6db-673a-44f7-b199-571d7021e70e","sink-uri":"pulsar://127.0.0.1:6650/ticdc-processor-resolved-ts-fallback-test-18884?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:50:21.314118636+08:00","start-ts":449575969295171587,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575969439350785}+ grep -q 'failed to get info:' /tidb/cdc/default/default/changefeed/status/896ad6db-673a-44f7-b199-571d7021e70e {"checkpoint-ts":449575971116810251,"min-table-barrier-ts":449575971116810251,"admin-job-type":0} /tidb/cdc/default/default/task/position/259d6627-52bc-491e-bab7-3c670bf8e816/896ad6db-673a-44f7-b199-571d7021e70e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/e2c60e32-b70c-4e3c-952e-076e8ac6e346/896ad6db-673a-44f7-b199-571d7021e70e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852582901900235 {"id":7365852582901900235,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/896ad6db-673a-44f7-b199-571d7021e70e {UpstreamID:7365852582901900235 Namespace:default ID:896ad6db-673a-44f7-b199-571d7021e70e SinkURI:pulsar://127.0.0.1:6650/ticdc-processor-resolved-ts-fallback-test-18884?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:50:21.314118636 +0800 CST StartTs:449575969295171587 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002640ea0 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575969439350785} {CheckpointTs:449575971116810251 MinTableBarrierTs:449575971116810251 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/259d6627-52bc-491e-bab7-3c670bf8e816 {"id":"259d6627-52bc-491e-bab7-3c670bf8e816","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996211} /tidb/cdc/default/__cdc_meta__/capture/e2c60e32-b70c-4e3c-952e-076e8ac6e346 {"id":"e2c60e32-b70c-4e3c-952e-076e8ac6e346","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996225} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd32d631 259d6627-52bc-491e-bab7-3c670bf8e816 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd32d6d0 e2c60e32-b70c-4e3c-952e-076e8ac6e346 /tidb/cdc/default/default/changefeed/info/896ad6db-673a-44f7-b199-571d7021e70e {"upstream-id":7365852582901900235,"namespace":"default","changefeed-id":"896ad6db-673a-44f7-b199-571d7021e70e","sink-uri":"pulsar://127.0.0.1:6650/ticdc-processor-resolved-ts-fallback-test-18884?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:50:21.314118636+08:00","start-ts":449575969295171587,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575969439350785} /tidb/cdc/default/default/changefeed/status/896ad6db-673a-44f7-b199-571d7021e70e {"checkpoint-ts":449575971116810251,"min-table-barrier-ts":449575971116810251,"admin-job-type":0} /tidb/cdc/default/default/task/position/259d6627-52bc-491e-bab7-3c670bf8e816/896ad6db-673a-44f7-b199-571d7021e70e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/e2c60e32-b70c-4e3c-952e-076e8ac6e346/896ad6db-673a-44f7-b199-571d7021e70e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852582901900235 {"id":7365852582901900235,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x table batch_add_table.finish_mark not exists for 2-th check, retry later table processor_resolved_ts_fallback.t2 not exists for 1-th check, retry later + set +x [Mon May 6 19:50:28 CST 2024] <<<<<< START Pulsar consumer in owner_resign case >>>>>> cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' table ddl_manager.finish_mark not exists for 49-th check, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) 1 run task successfully table owner_resign.t1 not exists for 1-th check, retry later + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 check diff successfully check diff successfully wait process cdc.test exit for 1-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > wait process cdc.test exit for 1-th time... check diff failed 1-th time, retry later table batch_add_table.finish_mark exists check diff successfully wait process cdc.test exit for 2-th time... < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:50:30 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7d11868e-85ff-41d9-8a9d-c17ad0ff3c08 {"id":"7d11868e-85ff-41d9-8a9d-c17ad0ff3c08","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996227} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd6b4c32 7d11868e-85ff-41d9-8a9d-c17ad0ff3c08 /tidb/cdc/default/default/upstream/7365852650287886083 {"id":7365852650287886083,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7d11868e-85ff-41d9-8a9d-c17ad0ff3c08 {"id":"7d11868e-85ff-41d9-8a9d-c17ad0ff3c08","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996227} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd6b4c32 7d11868e-85ff-41d9-8a9d-c17ad0ff3c08 /tidb/cdc/default/default/upstream/7365852650287886083 {"id":7365852650287886083,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7d11868e-85ff-41d9-8a9d-c17ad0ff3c08 {"id":"7d11868e-85ff-41d9-8a9d-c17ad0ff3c08","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996227} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd6b4c32 7d11868e-85ff-41d9-8a9d-c17ad0ff3c08 /tidb/cdc/default/default/upstream/7365852650287886083 + grep -q 'etcd info' {"id":7365852650287886083,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x ++ date + echo '[Mon May 6 19:50:30 CST 2024] <<<<<< START pulsar cluster in event_filter case >>>>>>' [Mon May 6 19:50:30 CST 2024] <<<<<< START pulsar cluster in event_filter case >>>>>> + workdir=/tmp/tidb_cdc_test/event_filter + cluster_type=normal + cd /tmp/tidb_cdc_test/event_filter + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/event_filter/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/event_filter/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/event_filter/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/event_filter/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/event_filter/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/event_filter/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/event_filter/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/event_filter/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/event_filter/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/event_filter/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/event_filter/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/event_filter/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:50:30 CST 2024] <<<<<< START pulsar cluster in normal mode in event_filter case >>>>>>' [Mon May 6 19:50:30 CST 2024] <<<<<< START pulsar cluster in normal mode in event_filter case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/event_filter/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/event_filter/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/event_filter/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... table ddl_manager.finish_mark not exists for 50-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table owner_resign.t1 exists wait process cdc.test exit for 3-th time... + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public table processor_resolved_ts_fallback.t2 exists table processor_resolved_ts_fallback.t3 not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... check diff failed 1-th time, retry later wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:50:31 CST 2024] <<<<<< run test case capture_session_done_during_task success! >>>>>> wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:50:31 CST 2024] <<<<<< run test case tiflash success! >>>>>> check diff failed 2-th time, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:50:31 CST 2024] <<<<<< run test case batch_add_table success! >>>>>> + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 table processor_resolved_ts_fallback.t3 exists cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^2$' 2 run task successfully killing cdc1: 2703 cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused run task failed 1-th time, retry later table ddl_manager.finish_mark not exists for 51-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Tenant not found Reason: Tenant not found check diff successfully [Mon May 6 19:50:32 CST 2024] <<<<<< START cdc server in owner_resign case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerDontUpdateCheckpoint=return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.35373539.out server --log-file /tmp/tidb_cdc_test/owner_resign/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/owner_resign/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + i=1 + '[' 1 -gt 20 ']' + sleep 2 check diff failed 3-th time, retry later + nc -z localhost 6650 + i=3 + '[' 3 -gt 20 ']' + sleep 2 table ddl_manager.finish_mark not exists for 52-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused run task failed 2-th time, retry later + /usr/local/pulsar/bin/pulsar-admin namespaces list public + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:50:36 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/fcc4ed26-36b2-4416-bf3d-c61e2c6bd66b {UpstreamID:7365852581815843871 Namespace:default ID:fcc4ed26-36b2-4416-bf3d-c61e2c6bd66b SinkURI:pulsar://127.0.0.1:6650/ticdc-owner-resign-test-5698?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:50:26.917353142 +0800 CST StartTs:449575970731982856 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00155a6c0 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575970902376451} {CheckpointTs:449575973051957256 MinTableBarrierTs:449575973051957256 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/513d809e-f59f-49be-a420-be33f23ca54c {"id":"513d809e-f59f-49be-a420-be33f23ca54c","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996233} /tidb/cdc/default/__cdc_meta__/capture/7beb23e5-ad80-4c27-a174-147b06b1d9bb {"id":"7beb23e5-ad80-4c27-a174-147b06b1d9bb","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996211} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd345533 7beb23e5-ad80-4c27-a174-147b06b1d9bb /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd345611 513d809e-f59f-49be-a420-be33f23ca54c /tidb/cdc/default/default/changefeed/info/fcc4ed26-36b2-4416-bf3d-c61e2c6bd66b {"upstream-id":7365852581815843871,"namespace":"default","changefeed-id":"fcc4ed26-36b2-4416-bf3d-c61e2c6bd66b","sink-uri":"pulsar://127.0.0.1:6650/ticdc-owner-resign-test-5698?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:50:26.917353142+08:00","start-ts":449575970731982856,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575970902376451} /tidb/cdc/default/default/changefeed/status/fcc4ed26-36b2-4416-bf3d-c61e2c6bd66b {"checkpoint-ts":449575973051957256,"min-table-barrier-ts":449575973051957256,"admin-job-type":0} /tidb/cdc/default/default/task/position/513d809e-f59f-49be-a420-be33f23ca54c/fcc4ed26-36b2-4416-bf3d-c61e2c6bd66b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/7beb23e5-ad80-4c27-a174-147b06b1d9bb/fcc4ed26-36b2-4416-bf3d-c61e2c6bd66b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852581815843871 {"id":7365852581815843871,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/fcc4ed26-36b2-4416-bf3d-c61e2c6bd66b {UpstreamID:7365852581815843871 Namespace:default ID:fcc4ed26-36b2-4416-bf3d-c61e2c6bd66b SinkURI:pulsar://127.0.0.1:6650/ticdc-owner-resign-test-5698?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:50:26.917353142 +0800 CST StartTs:449575970731982856 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00155a6c0 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575970902376451} {CheckpointTs:449575973051957256 MinTableBarrierTs:449575973051957256 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/513d809e-f59f-49be-a420-be33f23ca54c {"id":"513d809e-f59f-49be-a420-be33f23ca54c","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996233} /tidb/cdc/default/__cdc_meta__/capture/7beb23e5-ad80-4c27-a174-147b06b1d9bb {"id":"7beb23e5-ad80-4c27-a174-147b06b1d9bb","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996211} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd345533 7beb23e5-ad80-4c27-a174-147b06b1d9bb /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd345611 513d809e-f59f-49be-a420-be33f23ca54c /tidb/cdc/default/default/changefeed/info/fcc4ed26-36b2-4416-bf3d-c61e2c6bd66b {"upstream-id":7365852581815843871,"namespace":"default","changefeed-id":"fcc4ed26-36b2-4416-bf3d-c61e2c6bd66b","sink-uri":"pulsar://127.0.0.1:6650/ticdc-owner-resign-test-5698?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:50:26.917353142+08:00","start-ts":449575970731982856,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575970902376451} /tidb/cdc/default/default/changefeed/status/fcc4ed26-36b2-4416-bf3d-c61e2c6bd66b {"checkpoint-ts":449575973051957256,"min-table-barrier-ts":449575973051957256,"admin-job-type":0} /tidb/cdc/default/default/task/position/513d809e-f59f-49be-a420-be33f23ca54c/fcc4ed26-36b2-4416-bf3d-c61e2c6bd66b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/7beb23e5-ad80-4c27-a174-147b06b1d9bb/fcc4ed26-36b2-4416-bf3d-c61e2c6bd66b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852581815843871 {"id":7365852581815843871,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/fcc4ed26-36b2-4416-bf3d-c61e2c6bd66b {UpstreamID:7365852581815843871 Namespace:default ID:fcc4ed26-36b2-4416-bf3d-c61e2c6bd66b SinkURI:pulsar://127.0.0.1:6650/ticdc-owner-resign-test-5698?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:50:26.917353142 +0800 CST StartTs:449575970731982856 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00155a6c0 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575970902376451} {CheckpointTs:449575973051957256 MinTableBarrierTs:449575973051957256 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/513d809e-f59f-49be-a420-be33f23ca54c {"id":"513d809e-f59f-49be-a420-be33f23ca54c","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996233} /tidb/cdc/default/__cdc_meta__/capture/7beb23e5-ad80-4c27-a174-147b06b1d9bb {"id":"7beb23e5-ad80-4c27-a174-147b06b1d9bb","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996211} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd345533 7beb23e5-ad80-4c27-a174-147b06b1d9bb /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd345611 513d809e-f59f-49be-a420-be33f23ca54c /tidb/cdc/default/default/changefeed/info/fcc4ed26-36b2-4416-bf3d-c61e2c6bd66b {"upstream-id":7365852581815843871,"namespace":"default","changefeed-id":"fcc4ed26-36b2-4416-bf3d-c61e2c6bd66b","sink-uri":"pulsar://127.0.0.1:6650/ticdc-owner-resign-test-5698?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:50:26.917353142+08:00","start-ts":449575970731982856,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575970902376451} /tidb/cdc/default/default/changefeed/status/fcc4ed26-36b2-4416-bf3d-c61e2c6bd66b {"checkpoint-ts":449575973051957256,"min-table-barrier-ts":449575973051957256,"admin-job-type":0} /tidb/cdc/default/default/task/position/513d809e-f59f-49be-a420-be33f23ca54c/fcc4ed26-36b2-4416-bf3d-c61e2c6bd66b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/7beb23e5-ad80-4c27-a174-147b06b1d9bb/fcc4ed26-36b2-4416-bf3d-c61e2c6bd66b {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852581815843871 {"id":7365852581815843871,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^2$' check diff failed 4-th time, retry later 2 run task successfully % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public table ddl_manager.finish_mark not exists for 53-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f7201c0016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jzh76, pid:8747, start at 2024-05-06 19:50:34.271079369 +0800 CST m=+5.683843386 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:34.279 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:34.247 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:34.247 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f7201c0016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jzh76, pid:8747, start at 2024-05-06 19:50:34.271079369 +0800 CST m=+5.683843386 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:34.279 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:34.247 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:34.247 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f721b00008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jzh76, pid:8818, start at 2024-05-06 19:50:34.355909741 +0800 CST m=+5.713416651 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:34.363 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:34.348 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:34.348 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_sequence/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_sequence/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff successfully public/default ++ date + echo '[Mon May 6 19:50:38 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:50:38 CST 2024] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.event_filter.cli.8969.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-event-filter-1102?protocol=canal-json&enable-tidb-extension=true' --server=127.0.0.1:8300 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/event_filter/conf/cf.toml public/default ++ date + echo '[Mon May 6 19:50:38 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:50:38 CST 2024] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.cli.10405.out cli changefeed create --start-ts=449575966826037249 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-multi-capture-test-4342?protocol=canal-json&enable-tidb-extension=true' --server=127.0.0.1:8301 table ddl_manager.finish_mark not exists for 54-th check, retry later cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused run task failed 3-th time, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.cli.10139.out cli tso query --pd=http://127.0.0.1:2379 check diff failed 1-th time, retry later Create changefeed successfully! ID: cdd89470-0826-45fc-b998-28c3d724ec36 Info: {"upstream_id":7365852650287886083,"namespace":"default","id":"cdd89470-0826-45fc-b998-28c3d724ec36","sink_uri":"pulsar://127.0.0.1:6650/ticdc-event-filter-1102?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:50:39.572601157+08:00","start_ts":449575974054133765,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["event_filter.*"],"event_filters":[{"matcher":["event_filter.t1"],"ignore_event":["drop table","delete"],"ignore_sql":null,"ignore_insert_value_expr":"id = 2 or city = 'tokyo'","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""},{"matcher":["event_filter.t_truncate"],"ignore_event":["truncate table"],"ignore_sql":null,"ignore_insert_value_expr":"","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""},{"matcher":["event_filter.t_alter"],"ignore_event":["alter table"],"ignore_sql":null,"ignore_insert_value_expr":"","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""}]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575974054133765,"checkpoint_ts":449575974054133765,"checkpoint_time":"2024-05-06 19:50:38.915"} PASS Create changefeed successfully! ID: e1dd4175-7074-4eed-a598-45647e1e5994 Info: {"upstream_id":7365852579471104142,"namespace":"default","id":"e1dd4175-7074-4eed-a598-45647e1e5994","sink_uri":"pulsar://127.0.0.1:6650/ticdc-multi-capture-test-4342?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:50:39.618502546+08:00","start_ts":449575966826037249,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575966826037249,"checkpoint_ts":449575966826037249,"checkpoint_time":"2024-05-06 19:50:11.342"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... coverage: 2.7% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 55-th check, retry later + set +x [Mon May 6 19:50:41 CST 2024] <<<<<< START Pulsar consumer in multi_capture case >>>>>> table multi_capture_1.usertable not exists for 1-th check, retry later + set +x [Mon May 6 19:50:41 CST 2024] <<<<<< START Pulsar consumer in event_filter case >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/ddl_attributes/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... [2024/05/06 19:50:31.752 +08:00] [INFO] [pd_service_discovery.go:1016] ["[pd] switch leader"] [new-leader=http://127.0.0.1:2379] [old-leader=] [2024/05/06 19:50:31.752 +08:00] [INFO] [pd_service_discovery.go:498] ["[pd] init cluster id"] [cluster-id=7365852080212050206] [2024/05/06 19:50:31.753 +08:00] [INFO] [client.go:606] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2024/05/06 19:50:31.753 +08:00] [INFO] [tso_client.go:236] ["[tso] switch dc tso global allocator serving url"] [dc-location=global] [new-url=http://127.0.0.1:2379] [2024/05/06 19:50:31.753 +08:00] [INFO] [tso_dispatcher.go:359] ["[tso] tso dispatcher created"] [dc-location=global] [2024/05/06 19:50:31.753 +08:00] [INFO] [client.go:612] ["[pd] service mode changed"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2024/05/06 19:50:31.755 +08:00] [INFO] [pd_service_discovery.go:1016] ["[pd] switch leader"] [new-leader=http://127.0.0.1:2379] [old-leader=] [2024/05/06 19:50:31.755 +08:00] [INFO] [pd_service_discovery.go:498] ["[pd] init cluster id"] [cluster-id=7365852080212050206] [2024/05/06 19:50:31.755 +08:00] [INFO] [client.go:606] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2024/05/06 19:50:31.755 +08:00] [INFO] [tso_client.go:236] ["[tso] switch dc tso global allocator serving url"] [dc-location=global] [new-url=http://127.0.0.1:2379] [2024/05/06 19:50:31.755 +08:00] [INFO] [tso_dispatcher.go:359] ["[tso] tso dispatcher created"] [dc-location=global] [2024/05/06 19:50:31.755 +08:00] [INFO] [client.go:612] ["[pd] service mode changed"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2024/05/06 19:50:31.756 +08:00] [INFO] [tikv_driver.go:197] ["using API V1."] [2024/05/06 19:50:31.756 +08:00] [INFO] [main.go:180] ["genLock started"] [2024/05/06 19:50:31.757 +08:00] [INFO] [store_cache.go:477] ["change store resolve state"] [store=2] [addr=127.0.0.1:20160] [from=unresolved] [to=resolved] [liveness-state=reachable] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/region_merge/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... [2024/05/06 19:50:41.757 +08:00] [INFO] [main.go:196] ["genLock done"] [2024/05/06 19:50:41.757 +08:00] [INFO] [pd_service_discovery.go:550] ["[pd] exit member loop due to context canceled"] [2024/05/06 19:50:41.757 +08:00] [INFO] [tso_dispatcher.go:214] ["exit tso requests cancel loop"] [2024/05/06 19:50:41.757 +08:00] [INFO] [tso_dispatcher.go:455] ["[tso] stop fetching the pending tso requests due to context canceled"] [dc-location=global] [2024/05/06 19:50:41.757 +08:00] [INFO] [tso_dispatcher.go:380] ["[tso] exit tso dispatcher"] [dc-location=global] [2024/05/06 19:50:41.757 +08:00] [INFO] [tso_batch_controller.go:158] ["[pd] clear the tso batch controller"] [max-batch-size=10000] [best-batch-size=1] [collected-request-count=0] [pending-request-count=0] [2024/05/06 19:50:41.757 +08:00] [INFO] [resource_manager_client.go:295] ["[resource manager] exit resource token dispatcher"] [2024/05/06 19:50:41.757 +08:00] [INFO] [tso_client.go:140] ["closing tso client"] [2024/05/06 19:50:41.757 +08:00] [INFO] [tso_dispatcher.go:268] ["exit tso dispatcher loop"] [2024/05/06 19:50:41.757 +08:00] [INFO] [tso_client.go:145] ["close tso client"] [2024/05/06 19:50:41.757 +08:00] [INFO] [tso_batch_controller.go:158] ["[pd] clear the tso batch controller"] [max-batch-size=10000] [best-batch-size=1] [collected-request-count=0] [pending-request-count=0] [2024/05/06 19:50:41.757 +08:00] [INFO] [tso_client.go:155] ["tso client is closed"] [2024/05/06 19:50:41.757 +08:00] [INFO] [pd_service_discovery.go:637] ["[pd] close pd service discovery client"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/multi_rocks/run.sh using Sink-Type: pulsar... <<================= + set +x + tso='449575974141427717 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575974141427717 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:50:41 CST 2024] <<<<<< START cdc server in ddl_sequence case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.1018010182.out server --log-file /tmp/tidb_cdc_test/ddl_sequence/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_sequence/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 2-th time, retry later check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 56-th check, retry later table multi_capture_1.usertable exists table multi_capture_2.usertable not exists for 1-th check, retry later check diff failed 2-th time, retry later table partition_table.t exists table partition_table.t1 exists table partition_table.t2 exists table partition_table.finish_mark not exists for 1-th check, retry later table event_filter.t1 does not exists table event_filter.t1 exists table event_filter.t_normal not exists for 1-th check, retry later check diff successfully % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 start tidb cluster in /tmp/tidb_cdc_test/ddl_attributes Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/region_merge Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... \033[0;36m<<< Run all test success >>>\033[0m + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:50:44 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ee28baad-3cec-4c8c-8a13-9a8cf39be830 {"id":"ee28baad-3cec-4c8c-8a13-9a8cf39be830","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996242} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd9dd242 ee28baad-3cec-4c8c-8a13-9a8cf39be830 /tidb/cdc/default/default/upstream/7365852694226786975 {"id":7365852694226786975,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ee28baad-3cec-4c8c-8a13-9a8cf39be830 {"id":"ee28baad-3cec-4c8c-8a13-9a8cf39be830","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996242} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd9dd242 ee28baad-3cec-4c8c-8a13-9a8cf39be830 /tidb/cdc/default/default/upstream/7365852694226786975 {"id":7365852694226786975,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ee28baad-3cec-4c8c-8a13-9a8cf39be830 {"id":"ee28baad-3cec-4c8c-8a13-9a8cf39be830","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996242} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbd9dd242 ee28baad-3cec-4c8c-8a13-9a8cf39be830 /tidb/cdc/default/default/upstream/7365852694226786975 {"id":7365852694226786975,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:50:44 CST 2024] <<<<<< START pulsar cluster in ddl_sequence case >>>>>>' [Mon May 6 19:50:44 CST 2024] <<<<<< START pulsar cluster in ddl_sequence case >>>>>> + workdir=/tmp/tidb_cdc_test/ddl_sequence + cluster_type=normal + cd /tmp/tidb_cdc_test/ddl_sequence + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_sequence/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/ddl_sequence/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/ddl_sequence/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_sequence/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/ddl_sequence/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/ddl_sequence/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/ddl_sequence/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/ddl_sequence/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_sequence/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/ddl_sequence/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/ddl_sequence/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/ddl_sequence/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:50:44 CST 2024] <<<<<< START pulsar cluster in normal mode in ddl_sequence case >>>>>>' [Mon May 6 19:50:44 CST 2024] <<<<<< START pulsar cluster in normal mode in ddl_sequence case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/ddl_sequence/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/ddl_sequence/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/ddl_sequence/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } table multi_capture_2.usertable exists table multi_capture_3.usertable not exists for 1-th check, retry later [Pipeline] // dir [Pipeline] } table ddl_manager.finish_mark not exists for 57-th check, retry later table partition_table.finish_mark not exists for 2-th check, retry later [Pipeline] // timeout [Pipeline] } cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' 1 run task successfully check diff failed 1-th time, retry later [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } table event_filter.t_normal exists table event_filter.t_truncate not exists for 1-th check, retry later [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } Verifying downstream PD is started... [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } check diff failed 3-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 table multi_capture_3.usertable exists table multi_capture_4.usertable not exists for 1-th check, retry later Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table ddl_manager.finish_mark not exists for 58-th check, retry later table partition_table.finish_mark not exists for 3-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... table test.t2 not exists for 1-th check, retry later check diff failed 4-th time, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... table event_filter.t_truncate exists table event_filter.t_alter not exists for 1-th check, retry later + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public table multi_capture_4.usertable exists wait process cdc.test exit for 3-th time... table ddl_manager.finish_mark not exists for 59-th check, retry later check diff failed 1-th time, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:50:49 CST 2024] <<<<<< run test case processor_resolved_ts_fallback success! >>>>>> table test.t2 not exists for 2-th check, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.finish_mark not exists for 4-th check, retry later check diff failed 5-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table event_filter.t_alter exists table test.t2 not exists for 3-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 60-th check, retry later check diff successfully ***************** properties ***************** "readproportion"="0" "threadcount"="2" "insertproportion"="0" "mysql.user"="root" "workload"="core" "mysql.db"="multi_capture_1" "recordcount"="20" "readallfields"="true" "updateproportion"="0" "mysql.port"="4000" "dotransactions"="false" "requestdistribution"="uniform" "scanproportion"="0" "operationcount"="0" "mysql.host"="127.0.0.1" ********************************************** Run finished, takes 21.397402ms INSERT - Takes(s): 0.0, Count: 19, OPS: 1438.7, Avg(us): 1768, Min(us): 504, Max(us): 8124, 95th(us): 9000, 99th(us): 9000 ***************** properties ***************** "mysql.user"="root" "recordcount"="20" "mysql.db"="multi_capture_2" "readallfields"="true" "threadcount"="2" "updateproportion"="0" "operationcount"="0" "requestdistribution"="uniform" "mysql.port"="4000" "workload"="core" "dotransactions"="false" "readproportion"="0" "scanproportion"="0" "insertproportion"="0" "mysql.host"="127.0.0.1" ********************************************** Run finished, takes 9.961977ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2267.8, Avg(us): 942, Min(us): 430, Max(us): 1595, 95th(us): 2000, 99th(us): 2000 ***************** properties ***************** "insertproportion"="0" "requestdistribution"="uniform" "threadcount"="2" "workload"="core" "dotransactions"="false" "readproportion"="0" "scanproportion"="0" "readallfields"="true" "mysql.port"="4000" "updateproportion"="0" "mysql.user"="root" "recordcount"="20" "operationcount"="0" "mysql.host"="127.0.0.1" "mysql.db"="multi_capture_3" ********************************************** Run finished, takes 10.731573ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2073.8, Avg(us): 929, Min(us): 488, Max(us): 1713, 95th(us): 2000, 99th(us): 2000 ***************** properties ***************** "readproportion"="0" "scanproportion"="0" "mysql.host"="127.0.0.1" "operationcount"="0" "workload"="core" "readallfields"="true" "requestdistribution"="uniform" "threadcount"="2" "mysql.port"="4000" "mysql.db"="multi_capture_4" "recordcount"="20" "insertproportion"="0" "updateproportion"="0" "mysql.user"="root" "dotransactions"="false" ********************************************** Run finished, takes 10.50526ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2191.1, Avg(us): 987, Min(us): 476, Max(us): 1628, 95th(us): 2000, 99th(us): 2000 check diff failed 1-th time, retry later table partition_table.finish_mark not exists for 5-th check, retry later table event_filter.finish_mark exists check diff failed 1-th time, retry later check diff successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) public/default ++ date + echo '[Mon May 6 19:50:52 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:50:52 CST 2024] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.cli.10631.out cli changefeed create --start-ts=449575974141427717 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-ddl-sequence-test-8272?protocol=canal-json&enable-tidb-extension=true' ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 61-th check, retry later check diff successfully table partition_table.finish_mark not exists for 6-th check, retry later Create changefeed successfully! ID: 0c72f2a6-dcec-4cd8-9e48-2bc91588ae6c Info: {"upstream_id":7365852694226786975,"namespace":"default","id":"0c72f2a6-dcec-4cd8-9e48-2bc91588ae6c","sink_uri":"pulsar://127.0.0.1:6650/ticdc-ddl-sequence-test-8272?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:50:53.875509666+08:00","start_ts":449575974141427717,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575974141427717,"checkpoint_ts":449575974141427717,"checkpoint_time":"2024-05-06 19:50:39.248"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... table test.t2 not exists for 4-th check, retry later wait process cdc.test exit for 1-th time... check diff successfully check diff failed 2-th time, retry later wait process cdc.test exit for 2-th time... wait process cdc.test exit for 1-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f85804000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-qx8wr, pid:4497, start at 2024-05-06 19:50:54.225034905 +0800 CST m=+5.705946910 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:54.232 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:54.209 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:54.209 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x [Mon May 6 19:50:55 CST 2024] <<<<<< START Pulsar consumer in ddl_sequence case >>>>>> wait process cdc.test exit for 3-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f85ca80019 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-txbrj, pid:4710, start at 2024-05-06 19:50:54.53946841 +0800 CST m=+5.159486663 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:54.546 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:54.506 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:54.506 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f85ca80019 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-txbrj, pid:4710, start at 2024-05-06 19:50:54.53946841 +0800 CST m=+5.159486663 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:54.546 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:54.506 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:54.506 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f85d4c0015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-txbrj, pid:4803, start at 2024-05-06 19:50:54.595963806 +0800 CST m=+5.162523557 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:54.602 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:54.596 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:54.596 +0800 All versions after safe point can be accessed. (DO NOT EDIT) wait process cdc.test exit for 2-th time... table ddl_manager.finish_mark not exists for 62-th check, retry later table partition_table.finish_mark not exists for 7-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:50:55 CST 2024] <<<<<< run test case multi_capture success! >>>>>> Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_attributes/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_attributes/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 3-th time... table test.t2 not exists for 5-th check, retry later check diff successfully cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:50:56 CST 2024] <<<<<< run test case owner_resign success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f85804000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-qx8wr, pid:4497, start at 2024-05-06 19:50:54.225034905 +0800 CST m=+5.705946910 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:54.232 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:54.209 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:54.209 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36f85894000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-qx8wr, pid:4581, start at 2024-05-06 19:50:54.255886101 +0800 CST m=+5.688576931 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:52:54.268 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:50:54.245 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:40:54.245 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/region_merge/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/region_merge/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 1-th time... check diff failed 1-th time, retry later wait process cdc.test exit for 2-th time... table ddl_manager.finish_mark not exists for 63-th check, retry later table partition_table.finish_mark exists + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.cli.6081.out cli tso query --pd=http://127.0.0.1:2379 check diff successfully table test.t2 not exists for 6-th check, retry later wait process cdc.test exit for 3-th time... wait process cdc.test exit for 1-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:50:58 CST 2024] <<<<<< run test case event_filter success! >>>>>> check diff failed 2-th time, retry later wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... [Mon May 6 19:50:58 CST 2024] <<<<<< START cdc server in region_merge case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.region_merge.59665968.out server --log-file /tmp/tidb_cdc_test/region_merge/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/region_merge/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 64-th check, retry later + set +x + tso='449575979058724865 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575979058724865 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:50:59 CST 2024] <<<<<< START cdc server in ddl_attributes case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.61206122.out server --log-file /tmp/tidb_cdc_test/ddl_attributes/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_attributes/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:50:59 CST 2024] <<<<<< run test case partition_table success! >>>>>> table test.t2 not exists for 7-th check, retry later check diff failed 3-th time, retry later table test.t2 exists \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } table ddl_manager.finish_mark not exists for 65-th check, retry later Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc already exists) table ddl_sequence.finish_mark not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:51:01 GMT < Content-Length: 872 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/28b3f6c5-9909-4ef6-8a2a-656d714536e3 {"id":"28b3f6c5-9909-4ef6-8a2a-656d714536e3","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996259} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbdefb071 28b3f6c5-9909-4ef6-8a2a-656d714536e3 /tidb/cdc/default/default/upstream/7365852789212977999 {"id":7365852789212977999,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/28b3f6c5-9909-4ef6-8a2a-656d714536e3 {"id":"28b3f6c5-9909-4ef6-8a2a-656d714536e3","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996259} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbdefb071 28b3f6c5-9909-4ef6-8a2a-656d714536e3 /tidb/cdc/default/default/upstream/7365852789212977999 {"id":7365852789212977999,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/28b3f6c5-9909-4ef6-8a2a-656d714536e3 {"id":"28b3f6c5-9909-4ef6-8a2a-656d714536e3","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996259} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbdefb071 28b3f6c5-9909-4ef6-8a2a-656d714536e3 /tidb/cdc/default/default/upstream/7365852789212977999 {"id":7365852789212977999,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:51:01 CST 2024] <<<<<< START pulsar cluster in region_merge case >>>>>>' [Mon May 6 19:51:01 CST 2024] <<<<<< START pulsar cluster in region_merge case >>>>>> + workdir=/tmp/tidb_cdc_test/region_merge + cluster_type=mtls + cd /tmp/tidb_cdc_test/region_merge + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/region_merge/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/region_merge/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/region_merge/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/region_merge/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/region_merge/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/region_merge/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/region_merge/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/region_merge/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/region_merge/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/region_merge/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/region_merge/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/region_merge/pulsar_standalone.conf + pulsar_port=6650 + '[' mtls == mtls ']' + pulsar_port=6651 + gen_mtls_config + openssl genrsa -out ca.key.pem 2048 Generating RSA private key, 2048 bit long modulus .........+++ ...................................................................+++ e is 65537 (0x10001) check diff successfully [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } + openssl req -x509 -new -nodes -key ca.key.pem -subj /CN=CARoot -days 365 -out ca.cert.pem + openssl genrsa -out server.key.pem 2048 Generating RSA private key, 2048 bit long modulus .........................+++ ..+++ e is 65537 (0x10001) + openssl pkcs8 -topk8 -inform PEM -outform PEM -in server.key.pem -out server.key-pk8.pem -nocrypt + echo '[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + openssl req -new -config server.conf -key server.key.pem -out server.csr.pem -sha256 + openssl x509 -req -in server.csr.pem -CA ca.cert.pem -CAkey ca.key.pem -CAcreateserial -out server.cert.pem -days 365 -extensions v3_ext -extfile server.conf -sha256 Signature ok subject=/CN=server Getting CA Private Key + openssl genrsa -out broker_client.key.pem 2048 Generating RSA private key, 2048 bit long modulus ........................................+++ .......................................................+++ e is 65537 (0x10001) + openssl pkcs8 -topk8 -inform PEM -outform PEM -in broker_client.key.pem -out broker_client.key-pk8.pem -nocrypt + openssl req -new -subj /CN=broker_client -key broker_client.key.pem -out broker_client.csr.pem -sha256 + openssl x509 -req -in broker_client.csr.pem -CA ca.cert.pem -CAkey ca.key.pem -CAcreateserial -out broker_client.cert.pem -days 365 -sha256 Signature ok subject=/CN=broker_client Getting CA Private Key + echo ' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/region_merge/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/region_merge/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/region_merge/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/region_merge/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/region_merge/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/region_merge/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' + echo ' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/region_merge/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/region_merge/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/region_merge/ca.cert.pem' ++ date + echo '[Mon May 6 19:51:02 CST 2024] <<<<<< START pulsar cluster in mtls mode in region_merge case >>>>>>' [Mon May 6 19:51:02 CST 2024] <<<<<< START pulsar cluster in mtls mode in region_merge case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6651 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/region_merge/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/region_merge/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/region_merge/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv wait process cdc.test exit for 1-th time... [Pipeline] } [Pipeline] // node check diff failed 4-th time, retry later [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:51:02 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/19f6c3fd-636d-49fb-a4c8-77ea111d4da7 {"id":"19f6c3fd-636d-49fb-a4c8-77ea111d4da7","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996259} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbdef1a35 19f6c3fd-636d-49fb-a4c8-77ea111d4da7 /tidb/cdc/default/default/upstream/7365852788742989165 {"id":7365852788742989165,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/19f6c3fd-636d-49fb-a4c8-77ea111d4da7 {"id":"19f6c3fd-636d-49fb-a4c8-77ea111d4da7","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996259} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbdef1a35 19f6c3fd-636d-49fb-a4c8-77ea111d4da7 /tidb/cdc/default/default/upstream/7365852788742989165 {"id":7365852788742989165,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/19f6c3fd-636d-49fb-a4c8-77ea111d4da7 {"id":"19f6c3fd-636d-49fb-a4c8-77ea111d4da7","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996259} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbdef1a35 19f6c3fd-636d-49fb-a4c8-77ea111d4da7 /tidb/cdc/default/default/upstream/7365852788742989165 {"id":7365852788742989165,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:51:02 CST 2024] <<<<<< START pulsar cluster in ddl_attributes case >>>>>>' [Mon May 6 19:51:02 CST 2024] <<<<<< START pulsar cluster in ddl_attributes case >>>>>> + workdir=/tmp/tidb_cdc_test/ddl_attributes + cluster_type=normal + cd /tmp/tidb_cdc_test/ddl_attributes + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_attributes/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/ddl_attributes/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/ddl_attributes/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_attributes/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/ddl_attributes/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/ddl_attributes/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/ddl_attributes/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/ddl_attributes/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_attributes/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/ddl_attributes/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/ddl_attributes/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/ddl_attributes/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:51:02 CST 2024] <<<<<< START pulsar cluster in normal mode in ddl_attributes case >>>>>>' [Mon May 6 19:51:02 CST 2024] <<<<<< START pulsar cluster in normal mode in ddl_attributes case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/ddl_attributes/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/ddl_attributes/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/ddl_attributes/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 [Pipeline] // stage [Pipeline] } wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... table ddl_manager.finish_mark not exists for 66-th check, retry later table ddl_sequence.finish_mark not exists for 2-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:51:03 CST 2024] <<<<<< run test case resolve_lock success! >>>>>> + nc -z localhost 6651 + i=2 + '[' 2 -gt 20 ']' + sleep 2 + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 check diff failed 5-th time, retry later table ddl_manager.finish_mark not exists for 67-th check, retry later table ddl_sequence.finish_mark not exists for 3-th check, retry later + nc -z localhost 6651 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public check diff successfully table ddl_manager.finish_mark not exists for 68-th check, retry later table ddl_sequence.finish_mark not exists for 4-th check, retry later Tenant not found Reason: Tenant not found Tenant not found Reason: Tenant not found + i=1 + '[' 1 -gt 20 ']' + sleep 2 + i=1 + '[' 1 -gt 20 ']' + sleep 2 table ddl_sequence.finish_mark exists \033[0;36m<<< Run all test success >>>\033[0m check diff successfully [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } table ddl_manager.finish_mark not exists for 69-th check, retry later [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv wait process cdc.test exit for 1-th time... [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/run.sh using Sink-Type: pulsar... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo + source /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/synced_status_with_redo + CDC_BINARY=cdc.test + SINK_TYPE=pulsar + CDC_COUNT=3 + DB_COUNT=4 + trap stop_tidb_cluster EXIT + run_normal_case_and_unavailable_pd conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status_with_redo + mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo The 1 times to try to start tidb cluster... [Pipeline] // withEnv =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/sql_mode/run.sh using Sink-Type: pulsar... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/sql_mode/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/sql_mode ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/sql_mode + source /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/sql_mode/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/sql_mode + CDC_BINARY=cdc.test + SINK_TYPE=pulsar + CDC_COUNT=3 + DB_COUNT=4 + rm -rf /tmp/tidb_cdc_test/sql_mode + mkdir -p /tmp/tidb_cdc_test/sql_mode + start_tidb_cluster --workdir /tmp/tidb_cdc_test/sql_mode The 1 times to try to start tidb cluster... [Pipeline] } [Pipeline] // stage [Pipeline] } wait process cdc.test exit for 2-th time... + /usr/local/pulsar/bin/pulsar-admin namespaces list public wait process cdc.test exit for 3-th time... check diff failed 1-th time, retry later + /usr/local/pulsar/bin/pulsar-admin namespaces list public =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/processor_etcd_worker_delay/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:51:11 CST 2024] <<<<<< run test case ddl_sequence success! >>>>>> table ddl_manager.finish_mark not exists for 70-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... public/default check diff failed 2-th time, retry later ++ date + echo '[Mon May 6 19:51:13 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:51:13 CST 2024] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.cli.6658.out cli changefeed create --start-ts=449575979058724865 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-ddl-attributes-test-21824?protocol=canal-json&enable-tidb-extension=true' public/default table ddl_manager.finish_mark not exists for 71-th check, retry later Create changefeed successfully! ID: 9c9e6d9e-3d1a-4269-b514-084ef0b54d5d Info: {"upstream_id":7365852788742989165,"namespace":"default","id":"9c9e6d9e-3d1a-4269-b514-084ef0b54d5d","sink_uri":"pulsar://127.0.0.1:6650/ticdc-ddl-attributes-test-21824?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:51:14.231020195+08:00","start_ts":449575979058724865,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575979058724865,"checkpoint_ts":449575979058724865,"checkpoint_time":"2024-05-06 19:50:58.006"} PASS ++ date + echo '[Mon May 6 19:51:14 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:51:14 CST 2024] <<<<<< pulsar is ready >>>>>> coverage: 2.6% of statements in github.com/pingcap/tiflow/... Create changefeed successfully! ID: 1a023be4-7f4a-4a42-99c0-9cb83d139332 Info: {"upstream_id":7365852789212977999,"namespace":"default","id":"1a023be4-7f4a-4a42-99c0-9cb83d139332","sink_uri":"pulsar+ssl://127.0.0.1:6651/ticdc-region-merge-test-22386?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:51:15.024441933+08:00","start_ts":449575983332720643,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"tls-trust-certs-file-path":"/tmp/tidb_cdc_test/region_merge/ca.cert.pem","connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30,"auth-tls-certificate-path":"/tmp/tidb_cdc_test/region_merge/broker_client.cert.pem","auth-tls-private-key-path":"/tmp/tidb_cdc_test/region_merge/broker_client.key-pk8.pem"},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575983332720643,"checkpoint_ts":449575983332720643,"checkpoint_time":"2024-05-06 19:51:14.310"} [Mon May 6 19:51:15 CST 2024] <<<<<< START Pulsar consumer in region_merge case >>>>>> split_and_random_merge scale: 20 check diff failed 3-th time, retry later + set +x [Mon May 6 19:51:15 CST 2024] <<<<<< START Pulsar consumer in ddl_attributes case >>>>>> Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release start tidb cluster in /tmp/tidb_cdc_test/processor_etcd_worker_delay Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... table ddl_manager.finish_mark not exists for 72-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/sql_mode Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... check diff failed 4-th time, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/move_table/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... table ddl_manager.finish_mark not exists for 73-th check, retry later check diff successfully Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/move_table Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... table ddl_manager.finish_mark exists check diff successfully Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... wait process cdc.test exit for 1-th time... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... *************************** 1. row *************************** count(distinct region_id): 1 wait process cdc.test exit for 3-th time... table ddl_attributes.attributes_t1_new not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:51:22 CST 2024] <<<<<< run test case ddl_manager success! >>>>>> Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table ddl_attributes.attributes_t1_new not exists for 2-th check, retry later check diff failed 1-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa1ccc0018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6, pid:11750, start at 2024-05-06 19:51:23.210262824 +0800 CST m=+5.177929481 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:23.217 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:23.187 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:23.187 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa1ccc0018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6, pid:11750, start at 2024-05-06 19:51:23.210262824 +0800 CST m=+5.177929481 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:23.217 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:23.187 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:23.187 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa1ca80015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6, pid:11843, start at 2024-05-06 19:51:23.222077869 +0800 CST m=+5.135934332 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:23.228 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:23.228 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:23.228 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_attributes.attributes_t1_new exists table ddl_attributes.finish_mark not exists for 1-th check, retry later check diff failed 2-th time, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) split_and_random_merge scale: 40 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/resourcecontrol/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... + cd /tmp/tidb_cdc_test/synced_status_with_redo ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.13174.out cli tso query --pd=http://127.0.0.1:2379 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 1-th time, retry later check data successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:51:24 CST 2024] <<<<<< run test case ddl_puller_lag success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_attributes.finish_mark not exists for 2-th check, retry later check diff failed 3-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa52d40016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-807c6, pid:4874, start at 2024-05-06 19:51:26.666238494 +0800 CST m=+5.173883059 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:26.675 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:26.645 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:26.645 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa52d40016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-807c6, pid:4874, start at 2024-05-06 19:51:26.666238494 +0800 CST m=+5.173883059 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:26.675 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:26.645 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:26.645 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa54400014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-807c6, pid:4954, start at 2024-05-06 19:51:26.765250717 +0800 CST m=+5.222899508 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:26.772 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:26.736 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:26.736 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + set +x + tso='449575986577014785 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575986577014785 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449575986577014785 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test [Mon May 6 19:51:28 CST 2024] <<<<<< START cdc server in synced_status_with_redo case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.1321213214.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 start tidb cluster in /tmp/tidb_cdc_test/resourcecontrol Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_attributes.finish_mark not exists for 3-th check, retry later check diff failed 4-th time, retry later ++ date + echo '[Mon May 6 19:51:29 CST 2024] <<<<<< START pulsar cluster in processor_etcd_worker_delay case >>>>>>' [Mon May 6 19:51:29 CST 2024] <<<<<< START pulsar cluster in processor_etcd_worker_delay case >>>>>> + workdir=/tmp/tidb_cdc_test/processor_etcd_worker_delay + cluster_type=normal + cd /tmp/tidb_cdc_test/processor_etcd_worker_delay + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/processor_etcd_worker_delay/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/processor_etcd_worker_delay/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/processor_etcd_worker_delay/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/processor_etcd_worker_delay/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/processor_etcd_worker_delay/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/processor_etcd_worker_delay/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/processor_etcd_worker_delay/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/processor_etcd_worker_delay/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/processor_etcd_worker_delay/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/processor_etcd_worker_delay/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/processor_etcd_worker_delay/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/processor_etcd_worker_delay/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:51:29 CST 2024] <<<<<< START pulsar cluster in normal mode in processor_etcd_worker_delay case >>>>>>' [Mon May 6 19:51:29 CST 2024] <<<<<< START pulsar cluster in normal mode in processor_etcd_worker_delay case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/processor_etcd_worker_delay/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/processor_etcd_worker_delay/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/processor_etcd_worker_delay/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 Verifying downstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:51:31 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/56d146f9-69f9-4ef4-98ff-45ef5bcbf900 {"id":"56d146f9-69f9-4ef4-98ff-45ef5bcbf900","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996288} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe5d2e35 56d146f9-69f9-4ef4-98ff-45ef5bcbf900 /tidb/cdc/default/default/upstream/7365852913784148208 {"id":7365852913784148208,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/56d146f9-69f9-4ef4-98ff-45ef5bcbf900 {"id":"56d146f9-69f9-4ef4-98ff-45ef5bcbf900","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996288} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe5d2e35 56d146f9-69f9-4ef4-98ff-45ef5bcbf900 /tidb/cdc/default/default/upstream/7365852913784148208 {"id":7365852913784148208,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/56d146f9-69f9-4ef4-98ff-45ef5bcbf900 {"id":"56d146f9-69f9-4ef4-98ff-45ef5bcbf900","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996288} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe5d2e35 56d146f9-69f9-4ef4-98ff-45ef5bcbf900 /tidb/cdc/default/default/upstream/7365852913784148208 {"id":7365852913784148208,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449575986577014785 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.13270.out cli changefeed create --start-ts=449575986577014785 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml table ddl_attributes.finish_mark not exists for 4-th check, retry later check diff failed 5-th time, retry later Create changefeed successfully! ID: test-1 Info: {"upstream_id":7365852913784148208,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-06T19:51:31.710869881+08:00","start_ts":449575986577014785,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575986577014785,"checkpoint_ts":449575986577014785,"checkpoint_time":"2024-05-06 19:51:26.686"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 *************************** 1. row *************************** count(distinct region_id): 1 Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa6880000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh, pid:10224, start at 2024-05-06 19:51:28.044345546 +0800 CST m=+5.421921194 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:28.052 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:28.032 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:28.032 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa6880000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh, pid:10224, start at 2024-05-06 19:51:28.044345546 +0800 CST m=+5.421921194 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:28.052 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:28.032 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:28.032 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa69340014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh, pid:10300, start at 2024-05-06 19:51:28.121862676 +0800 CST m=+5.452124468 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:28.129 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:28.126 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:28.126 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sql_mode/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sql_mode/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/sql_mode/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/sql_mode/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sql_mode/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + set +x ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2035 0 --:--:-- --:--:-- --:--:-- 2046 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-06 19:51:26.686","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-05-06 19:51:33.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '19:51:26.686","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-06' '19:51:33.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '19:51:26.686","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-06' '19:51:33.000","info":"Data' syncing is 'finished"}' ++ jq -r .sink_checkpoint_ts + sink_checkpoint_ts='2024-05-06 19:51:26.686' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '19:51:26.686","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-06' '19:51:33.000","info":"Data' syncing is 'finished"}' ++ jq -r .puller_resolved_ts + puller_resolved_ts='1970-01-01 08:00:00.000' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '19:51:26.686","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-06' '19:51:33.000","info":"Data' syncing is 'finished"}' ++ jq -r .last_synced_ts table ddl_attributes.finish_mark not exists for 5-th check, retry later + last_synced_ts='1970-01-01 08:00:00.000' + '[' true '!=' true ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' ++ date '+%Y-%m-%d %H:%M:%S' + current='2024-05-06 19:51:33' + echo 'sink_checkpoint_ts is 2024-05-06' 19:51:26.686 sink_checkpoint_ts is 2024-05-06 19:51:26.686 ++ date -d '2024-05-06 19:51:26.686' +%s + checkpoint_timestamp=1714996286 ++ date -d '2024-05-06 19:51:33' +%s + current_timestamp=1714996293 + '[' 7 -gt 300 ']' + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa90440005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-v1jxz, pid:24142, start at 2024-05-06 19:51:30.580720226 +0800 CST m=+5.352531514 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:30.587 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:30.577 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:30.577 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa90440005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-v1jxz, pid:24142, start at 2024-05-06 19:51:30.580720226 +0800 CST m=+5.352531514 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:30.587 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:30.577 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:30.577 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa90ec0016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-v1jxz, pid:24222, start at 2024-05-06 19:51:30.65040388 +0800 CST m=+5.366024624 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:30.657 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:30.619 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:30.619 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/move_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/move_table/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/move_table/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/move_table/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/move_table/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff successfully Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/ddl_only_block_related_table/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/multi_source/run.sh using Sink-Type: pulsar... <<================= + trap stop_tidb_cluster EXIT + run_sql 'set global sql_mode='\''NO_BACKSLASH_ESCAPES'\'';' 127.0.0.1 4000 + run_sql 'set global sql_mode='\''NO_BACKSLASH_ESCAPES'\'';' 127.0.0.1 3306 + cd /tmp/tidb_cdc_test/sql_mode ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.cli.11701.out cli tso query --pd=http://127.0.0.1:2379 table ddl_attributes.finish_mark not exists for 6-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.cli.25495.out cli tso query --pd=http://127.0.0.1:2379 table test.t1 exists + sleep 5 + nc -z localhost 6650 + i=3 + '[' 3 -gt 20 ']' + sleep 2 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) split_and_random_merge scale: 80 + set +x + tso='449575988869726209 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575988869726209 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449575988869726209 + run_cdc_server --workdir /tmp/tidb_cdc_test/sql_mode --binary cdc.test [Mon May 6 19:51:36 CST 2024] <<<<<< START cdc server in sql_mode case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.1173511737.out server --log-file /tmp/tidb_cdc_test/sql_mode/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sql_mode/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_attributes.finish_mark not exists for 7-th check, retry later + set +x + tso='449575988986380290 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575988986380290 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x \033[0;36m<<< Run all test success >>>\033[0m start tidb cluster in /tmp/tidb_cdc_test/ddl_only_block_related_table Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... check diff failed 1-th time, retry later [Pipeline] } + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc already exists) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // cache ***************** properties ***************** "threadcount"="10" "updateproportion"="0" "requestdistribution"="uniform" "operationcount"="0" "mysql.port"="4000" "readallfields"="true" "mysql.db"="move_table" "scanproportion"="0" "readproportion"="0" "insertproportion"="0" "mysql.user"="root" "dotransactions"="false" "workload"="core" "mysql.host"="127.0.0.1" "recordcount"="10000" ********************************************** [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table ddl_attributes.finish_mark not exists for 8-th check, retry later check diff failed 2-th time, retry later public/default + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:51:40 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ed124ae1-ad47-4766-864b-52215fa9eaf5 {"id":"ed124ae1-ad47-4766-864b-52215fa9eaf5","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996297} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe6d303f ed124ae1-ad47-4766-864b-52215fa9eaf5 /tidb/cdc/default/default/upstream/7365852935194214761 {"id":7365852935194214761,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ed124ae1-ad47-4766-864b-52215fa9eaf5 {"id":"ed124ae1-ad47-4766-864b-52215fa9eaf5","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996297} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe6d303f ed124ae1-ad47-4766-864b-52215fa9eaf5 /tidb/cdc/default/default/upstream/7365852935194214761 {"id":7365852935194214761,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ed124ae1-ad47-4766-864b-52215fa9eaf5 {"id":"ed124ae1-ad47-4766-864b-52215fa9eaf5","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996297} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe6d303f ed124ae1-ad47-4766-864b-52215fa9eaf5 /tidb/cdc/default/default/upstream/7365852935194214761 {"id":7365852935194214761,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449575988869726209 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.cli.11794.out cli changefeed create --start-ts=449575988869726209 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 Run finished, takes 1.618486475s INSERT - Takes(s): 1.6, Count: 10000, OPS: 6198.6, Avg(us): 1570, Min(us): 846, Max(us): 29568, 95th(us): 2000, 99th(us): 20000 [Mon May 6 19:51:40 CST 2024] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.2558525587.out server --log-file /tmp/tidb_cdc_test/move_table/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data1 --cluster-id default --addr 127.0.0.1:8300 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ++ date + echo '[Mon May 6 19:51:40 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:51:40 CST 2024] <<<<<< pulsar is ready >>>>>> [Mon May 6 19:51:40 CST 2024] <<<<<< START cdc server in processor_etcd_worker_delay case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/orchestrator/ProcessorEtcdDelay=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_etcd_worker_delay.66406642.out server --log-file /tmp/tidb_cdc_test/processor_etcd_worker_delay/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_etcd_worker_delay/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' Create changefeed successfully! ID: test-1 Info: {"upstream_id":7365852935194214761,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-06T19:51:40.446266317+08:00","start_ts":449575988869726209,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575988869726209,"checkpoint_ts":449575988869726209,"checkpoint_time":"2024-05-06 19:51:35.432"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Verifying downstream PD is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fb28840012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jzh76, pid:11787, start at 2024-05-06 19:51:40.348335598 +0800 CST m=+6.001442263 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:40.356 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:40.321 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:40.321 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 2503 0 --:--:-- --:--:-- --:--:-- 2531 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-06 19:51:39.086","puller_resolved_ts":"2024-05-06 19:51:33.436","last_synced_ts":"2024-05-06 19:51:33.487","now_ts":"2024-05-06 19:51:40.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '19:51:39.086","puller_resolved_ts":"2024-05-06' '19:51:33.436","last_synced_ts":"2024-05-06' '19:51:33.487","now_ts":"2024-05-06' '19:51:40.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '19:51:39.086","puller_resolved_ts":"2024-05-06' '19:51:33.436","last_synced_ts":"2024-05-06' '19:51:33.487","now_ts":"2024-05-06' '19:51:40.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 table ddl_attributes.finish_mark not exists for 9-th check, retry later check diff failed 3-th time, retry later Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + set +x + run_sql 'use test; create table t1(id bigint primary key, a text, b text as ((regexp_replace(a, '\''^[1-9]\d{9,29}$'\'', '\''aaaaa'\''))), c text); insert into t1 (id, a, c) values(1,123456, '\''ab\\\\c'\''); insert into t1 (id, a, c) values(2,1234567890123, '\''ab\\c'\'');' 127.0.0.1 4000 + '[' pulsar == mysql ']' + stop_tidb_cluster VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fb28840012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jzh76, pid:11787, start at 2024-05-06 19:51:40.348335598 +0800 CST m=+6.001442263 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:40.356 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:40.321 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:40.321 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fb1fd80014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jzh76, pid:11865, start at 2024-05-06 19:51:39.791169105 +0800 CST m=+5.390530260 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:39.799 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:39.766 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:39.766 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/resourcecontrol/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/resourcecontrol/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } *************************** 1. row *************************** count(distinct region_id): 4 table ddl_attributes.finish_mark not exists for 10-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:51:43 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b6e9c580-caaa-4156-a79a-ba4f97d0e057 {"id":"b6e9c580-caaa-4156-a79a-ba4f97d0e057","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996300} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe7d2b58 b6e9c580-caaa-4156-a79a-ba4f97d0e057 /tidb/cdc/default/default/upstream/7365852946224692535 {"id":7365852946224692535,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b6e9c580-caaa-4156-a79a-ba4f97d0e057 {"id":"b6e9c580-caaa-4156-a79a-ba4f97d0e057","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996300} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe7d2b58 b6e9c580-caaa-4156-a79a-ba4f97d0e057 /tidb/cdc/default/default/upstream/7365852946224692535 {"id":7365852946224692535,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b6e9c580-caaa-4156-a79a-ba4f97d0e057 {"id":"b6e9c580-caaa-4156-a79a-ba4f97d0e057","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996300} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe7d2b58 b6e9c580-caaa-4156-a79a-ba4f97d0e057 /tidb/cdc/default/default/upstream/7365852946224692535 {"id":7365852946224692535,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:51:43 CST 2024] <<<<<< START pulsar cluster in move_table case >>>>>>' [Mon May 6 19:51:43 CST 2024] <<<<<< START pulsar cluster in move_table case >>>>>> + workdir=/tmp/tidb_cdc_test/move_table + cluster_type=normal + cd /tmp/tidb_cdc_test/move_table + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/move_table/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/move_table/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/move_table/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/move_table/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/move_table/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/move_table/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/move_table/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/move_table/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/move_table/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/move_table/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/move_table/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/move_table/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:51:43 CST 2024] <<<<<< START pulsar cluster in normal mode in move_table case >>>>>>' [Mon May 6 19:51:43 CST 2024] <<<<<< START pulsar cluster in normal mode in move_table case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/move_table/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/move_table/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/move_table/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:51:43 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/814d237a-2801-4881-9185-c60a2586ef5c {"id":"814d237a-2801-4881-9185-c60a2586ef5c","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996300} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe6ca659 814d237a-2801-4881-9185-c60a2586ef5c /tidb/cdc/default/default/upstream/7365852929774475396 {"id":7365852929774475396,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/814d237a-2801-4881-9185-c60a2586ef5c {"id":"814d237a-2801-4881-9185-c60a2586ef5c","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996300} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe6ca659 814d237a-2801-4881-9185-c60a2586ef5c /tidb/cdc/default/default/upstream/7365852929774475396 {"id":7365852929774475396,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/814d237a-2801-4881-9185-c60a2586ef5c {"id":"814d237a-2801-4881-9185-c60a2586ef5c","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996300} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe6ca659 814d237a-2801-4881-9185-c60a2586ef5c /tidb/cdc/default/default/upstream/7365852929774475396 {"id":7365852929774475396,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Mon May 6 19:51:44 CST 2024] <<<<<< START Pulsar consumer in processor_etcd_worker_delay case >>>>>> check diff failed 4-th time, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.cli.13195.out cli tso query --pd=http://127.0.0.1:2379 table ddl_attributes.finish_mark exists check diff successfully + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... check diff successfully wait process cdc.test exit for 2-th time... + set +x + tso='449575991383162881 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449575991383162881 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:51:46 CST 2024] <<<<<< START cdc server in resourcecontrol case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.1323113233.out server --log-file /tmp/tidb_cdc_test/resourcecontrol/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/resourcecontrol/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 1-th time... wait process cdc.test exit for 3-th time... wait process cdc.test exit for 2-th time... + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:51:47 CST 2024] <<<<<< run test case ddl_attributes success! >>>>>> wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:51:48 CST 2024] <<<<<< run test case changefeed_pause_resume success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table region_merge.t1 exists check diff failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:51:49 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fbb400000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-55h44, pid:7700, start at 2024-05-06 19:51:49.260964067 +0800 CST m=+5.343826577 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:49.270 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:49.248 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:49.248 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully wait process cdc.test exit for 1-th time... + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/281051f3-aa7e-42d6-bfa0-827e9b821bf1 {"id":"281051f3-aa7e-42d6-bfa0-827e9b821bf1","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996306} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe9ad53c 281051f3-aa7e-42d6-bfa0-827e9b821bf1 /tidb/cdc/default/default/upstream/7365852977337706292 {"id":7365852977337706292,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/281051f3-aa7e-42d6-bfa0-827e9b821bf1 {"id":"281051f3-aa7e-42d6-bfa0-827e9b821bf1","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996306} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe9ad53c 281051f3-aa7e-42d6-bfa0-827e9b821bf1 /tidb/cdc/default/default/upstream/7365852977337706292 {"id":7365852977337706292,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/281051f3-aa7e-42d6-bfa0-827e9b821bf1 {"id":"281051f3-aa7e-42d6-bfa0-827e9b821bf1","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996306} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe9ad53c 281051f3-aa7e-42d6-bfa0-827e9b821bf1 /tidb/cdc/default/default/upstream/7365852977337706292 {"id":7365852977337706292,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:51:50 CST 2024] <<<<<< START pulsar cluster in resourcecontrol case >>>>>>' [Mon May 6 19:51:50 CST 2024] <<<<<< START pulsar cluster in resourcecontrol case >>>>>> + workdir=/tmp/tidb_cdc_test/resourcecontrol + cluster_type=normal + cd /tmp/tidb_cdc_test/resourcecontrol + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/resourcecontrol/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/resourcecontrol/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/resourcecontrol/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/resourcecontrol/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/resourcecontrol/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/resourcecontrol/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/resourcecontrol/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/resourcecontrol/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/resourcecontrol/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/resourcecontrol/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/resourcecontrol/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/resourcecontrol/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:51:50 CST 2024] <<<<<< START pulsar cluster in normal mode in resourcecontrol case >>>>>>' [Mon May 6 19:51:50 CST 2024] <<<<<< START pulsar cluster in normal mode in resourcecontrol case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/resourcecontrol/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/resourcecontrol/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/resourcecontrol/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 wait process cdc.test exit for 2-th time... public/default ++ date + echo '[Mon May 6 19:51:51 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:51:51 CST 2024] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.cli.26040.out cli changefeed create --start-ts=449575988986380290 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-move-table-test-18591?protocol=canal-json&enable-tidb-extension=true' wait process cdc.test exit for 3-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fbb400000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-55h44, pid:7700, start at 2024-05-06 19:51:49.260964067 +0800 CST m=+5.343826577 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:49.270 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:49.248 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:49.248 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fbb4a80014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-55h44, pid:7782, start at 2024-05-06 19:51:49.330054483 +0800 CST m=+5.363305197 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:49.341 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:49.340 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:49.340 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Create changefeed successfully! ID: 3d1b9799-a1cb-4947-be8e-18c4a917b77e Info: {"upstream_id":7365852946224692535,"namespace":"default","id":"3d1b9799-a1cb-4947-be8e-18c4a917b77e","sink_uri":"pulsar://127.0.0.1:6650/ticdc-move-table-test-18591?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:51:52.132207376+08:00","start_ts":449575988986380290,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575988986380290,"checkpoint_ts":449575988986380290,"checkpoint_time":"2024-05-06 19:51:35.877"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:51:52 CST 2024] <<<<<< run test case region_merge success! >>>>>> + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 + set +x [Mon May 6 19:51:53 CST 2024] <<<<<< START Pulsar consumer in move_table case >>>>>> [Mon May 6 19:51:53 CST 2024] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.2610126107.out server --log-file /tmp/tidb_cdc_test/move_table/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data2 --cluster-id default --addr 127.0.0.1:8301 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Mon May 6 19:51:54 CST 2024] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.90489050.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + start_tidb_cluster --workdir /tmp/tidb_cdc_test/sql_mode The 1 times to try to start tidb cluster... + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:51:56 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/3d1b9799-a1cb-4947-be8e-18c4a917b77e {UpstreamID:7365852946224692535 Namespace:default ID:3d1b9799-a1cb-4947-be8e-18c4a917b77e SinkURI:pulsar://127.0.0.1:6650/ticdc-move-table-test-18591?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:51:52.132207376 +0800 CST StartTs:449575988986380290 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0025dc1b0 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575993246220289} {CheckpointTs:449575989654847580 MinTableBarrierTs:449575994347225089 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1bbea387-99f4-4f89-ba6b-5cceb39a9656 {"id":"1bbea387-99f4-4f89-ba6b-5cceb39a9656","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996314} /tidb/cdc/default/__cdc_meta__/capture/b6e9c580-caaa-4156-a79a-ba4f97d0e057 {"id":"b6e9c580-caaa-4156-a79a-ba4f97d0e057","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996300} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe7d2b58 b6e9c580-caaa-4156-a79a-ba4f97d0e057 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe7d2be5 1bbea387-99f4-4f89-ba6b-5cceb39a9656 /tidb/cdc/default/default/changefeed/info/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"upstream-id":7365852946224692535,"namespace":"default","changefeed-id":"3d1b9799-a1cb-4947-be8e-18c4a917b77e","sink-uri":"pulsar://127.0.0.1:6650/ticdc-move-table-test-18591?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:51:52.132207376+08:00","start-ts":449575988986380290,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575993246220289} /tidb/cdc/default/default/changefeed/status/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"checkpoint-ts":449575989654847580,"min-table-barrier-ts":449575994347225089,"admin-job-type":0} /tidb/cdc/default/default/task/position/1bbea387-99f4-4f89-ba6b-5cceb39a9656/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b6e9c580-caaa-4156-a79a-ba4f97d0e057/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852946224692535 {"id":7365852946224692535,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/3d1b9799-a1cb-4947-be8e-18c4a917b77e {UpstreamID:7365852946224692535 Namespace:default ID:3d1b9799-a1cb-4947-be8e-18c4a917b77e SinkURI:pulsar://127.0.0.1:6650/ticdc-move-table-test-18591?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:51:52.132207376 +0800 CST StartTs:449575988986380290 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0025dc1b0 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575993246220289} {CheckpointTs:449575989654847580 MinTableBarrierTs:449575994347225089 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1bbea387-99f4-4f89-ba6b-5cceb39a9656 {"id":"1bbea387-99f4-4f89-ba6b-5cceb39a9656","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996314} /tidb/cdc/default/__cdc_meta__/capture/b6e9c580-caaa-4156-a79a-ba4f97d0e057 {"id":"b6e9c580-caaa-4156-a79a-ba4f97d0e057","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996300} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe7d2b58 b6e9c580-caaa-4156-a79a-ba4f97d0e057 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe7d2be5 1bbea387-99f4-4f89-ba6b-5cceb39a9656 /tidb/cdc/default/default/changefeed/info/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"upstream-id":7365852946224692535,"namespace":"default","changefeed-id":"3d1b9799-a1cb-4947-be8e-18c4a917b77e","sink-uri":"pulsar://127.0.0.1:6650/ticdc-move-table-test-18591?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:51:52.132207376+08:00","start-ts":449575988986380290,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575993246220289} /tidb/cdc/default/default/changefeed/s+ grep -q 'failed to get info:' tatus/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"checkpoint-ts":449575989654847580,"min-table-barrier-ts":449575994347225089,"admin-job-type":0} /tidb/cdc/default/default/task/position/1bbea387-99f4-4f89-ba6b-5cceb39a9656/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b6e9c580-caaa-4156-a79a-ba4f97d0e057/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852946224692535 {"id":7365852946224692535,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/3d1b9799-a1cb-4947-be8e-18c4a917b77e {UpstreamID:7365852946224692535 Namespace:default ID:3d1b9799-a1cb-4947-be8e-18c4a917b77e SinkURI:pulsar://127.0.0.1:6650/ticdc-move-table-test-18591?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:51:52.132207376 +0800 CST StartTs:449575988986380290 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0025dc1b0 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575993246220289} {CheckpointTs:449575989654847580 MinTableBarrierTs:449575994347225089 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1bbea387-99f4-4f89-ba6b-5cceb39a9656 {"id":"1bbea387-99f4-4f89-ba6b-5cceb39a9656","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996314} /tidb/cdc/default/__cdc_meta__/capture/b6e9c580-caaa-4156-a79a-ba4f97d0e057 {"id":"b6e9c580-caaa-4156-a79a-ba4f97d0e057","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996300} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe7d2b58 b6e9c580-caaa-4156-a79a-ba4f97d0e057 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe7d2be5 1bbea387-99f4-4f89-ba6b-5cceb39a9656 /tidb/cdc/default/default/changefeed/info/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"upstream-id":7365852946224692535,"namespace":"default","changefeed-id":"3d1b9799-a1cb-4947-be8e-18c4a917b77e","sink-uri":"pulsar://127.0.0.1:6650/ticdc-move-table-test-18591?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:51:52.132207376+08:00","start-ts":449575988986380290,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575993246220289} /tidb/cdc/default/default/changefeed/status/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"checkpoint-ts":449575989654847580,"min-table-barrier-ts":449575994347225089,"admin-job-type":0} + grep -q 'etcd info' /tidb/cdc/default/default/task/position/1bbea387-99f4-4f89-ba6b-5cceb39a9656/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b6e9c580-caaa-4156-a79a-ba4f97d0e057/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852946224692535 {"id":7365852946224692535,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x [Mon May 6 19:51:56 CST 2024] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.2620926211.out server --log-file /tmp/tidb_cdc_test/move_table/cdc3.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data3 --cluster-id default --addr 127.0.0.1:8302 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Tenant not found Reason: Tenant not found + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:51:57 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3fbb9670-8952-4197-83f7-3ee3d1fa278c {"id":"3fbb9670-8952-4197-83f7-3ee3d1fa278c","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996314} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbec04037 3fbb9670-8952-4197-83f7-3ee3d1fa278c /tidb/cdc/default/default/upstream/7365853017852376063 {"id":7365853017852376063,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3fbb9670-8952-4197-83f7-3ee3d1fa278c {"id":"3fbb9670-8952-4197-83f7-3ee3d1fa278c","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996314} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbec04037 3fbb9670-8952-4197-83f7-3ee3d1fa278c /tidb/cdc/default/default/upstream/7365853017852376063 {"id":7365853017852376063,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3fbb9670-8952-4197-83f7-3ee3d1fa278c {"id":"3fbb9670-8952-4197-83f7-3ee3d1fa278c","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996314} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbec04037 3fbb9670-8952-4197-83f7-3ee3d1fa278c /tidb/cdc/default/default/upstream/7365853017852376063 {"id":7365853017852376063,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:51:57 CST 2024] <<<<<< START pulsar cluster in ddl_only_block_related_table case >>>>>>' [Mon May 6 19:51:57 CST 2024] <<<<<< START pulsar cluster in ddl_only_block_related_table case >>>>>> + workdir=/tmp/tidb_cdc_test/ddl_only_block_related_table + cluster_type=normal + cd /tmp/tidb_cdc_test/ddl_only_block_related_table + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_only_block_related_table/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/ddl_only_block_related_table/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/ddl_only_block_related_table/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_only_block_related_table/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/ddl_only_block_related_table/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/ddl_only_block_related_table/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/ddl_only_block_related_table/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/ddl_only_block_related_table/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/ddl_only_block_related_table/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/ddl_only_block_related_table/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/ddl_only_block_related_table/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/ddl_only_block_related_table/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:51:57 CST 2024] <<<<<< START pulsar cluster in normal mode in ddl_only_block_related_table case >>>>>>' [Mon May 6 19:51:57 CST 2024] <<<<<< START pulsar cluster in normal mode in ddl_only_block_related_table case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/ddl_only_block_related_table/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc already exists) + i=1 + '[' 1 -gt 20 ']' + sleep 2 [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/cli_with_auth/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... [Pipeline] } [Pipeline] // stage [Pipeline] } + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 + /usr/local/pulsar/bin/pulsar-admin namespaces list public start tidb cluster in /tmp/tidb_cdc_test/sql_mode Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:51:59 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/3d1b9799-a1cb-4947-be8e-18c4a917b77e {UpstreamID:7365852946224692535 Namespace:default ID:3d1b9799-a1cb-4947-be8e-18c4a917b77e SinkURI:pulsar://127.0.0.1:6650/ticdc-move-table-test-18591?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:51:52.132207376 +0800 CST StartTs:449575988986380290 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003eaa630 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575993246220289} {CheckpointTs:449575989698101339 MinTableBarrierTs:449575995133657091 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1bbea387-99f4-4f89-ba6b-5cceb39a9656 {"id":"1bbea387-99f4-4f89-ba6b-5cceb39a9656","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996314} /tidb/cdc/default/__cdc_meta__/capture/acc90939-d7b6-4bb8-a061-fb4993e3fafc {"id":"acc90939-d7b6-4bb8-a061-fb4993e3fafc","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996317} /tidb/cdc/default/__cdc_meta__/capture/b6e9c580-caaa-4156-a79a-ba4f97d0e057 {"id":"b6e9c580-caaa-4156-a79a-ba4f97d0e057","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996300} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe7d2b58 b6e9c580-caaa-4156-a79a-ba4f97d0e057 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe7d2be5 1bbea387-99f4-4f89-ba6b-5cceb39a9656 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe7d2c0f acc90939-d7b6-4bb8-a061-fb4993e3fafc /tidb/cdc/default/default/changefeed/info/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"upstream-id":7365852946224692535,"namespace":"default","changefeed-id":"3d1b9799-a1cb-4947-be8e-18c4a917b77e","sink-uri":"pulsar://127.0.0.1:6650/ticdc-move-table-test-18591?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:51:52.132207376+08:00","start-ts":449575988986380290,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575993246220289} /tidb/cdc/default/default/changefeed/status/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"checkpoint-ts":449575989698101339,"min-table-barrier-ts":449575995133657091,"admin-job-type":0} /tidb/cdc/default/default/task/position/1bbea387-99f4-4f89-ba6b-5cceb39a9656/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/acc90939-d7b6-4bb8-a061-fb4993e3fafc/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b6e9c580-caaa-4156-a79a-ba4f97d0e057/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852946224692535 {"id":7365852946224692535,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/3d1b9799-a1cb-4947-be8e-18c4a917b77e {UpstreamID:7365852946224692535 Namespace:default ID:3d1b9799-a1cb-4947-be8e-18c4a917b77e SinkURI:pulsar://127.0.0.1:6650/ticdc-move-table-test-18591?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:51:52.132207376 +0800 CST StartTs:449575988986380290 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003eaa630 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575993246220289} {CheckpointTs:449575989698101339 MinTableBarrierTs:449575995133657091 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1bbea387-99f4-4f89-ba6b-5cceb39a9656 {"id":"1bbea387-99f4-4f89-ba6b-5cceb39a9656","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996314} /tidb/cdc/default/__cdc_meta__/capture/acc90939-d7b6-4bb8-a061-fb4993e3fafc {"id":"acc90939-d7b6-4bb8-a061-fb4993e3fafc","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996317} /tidb/cdc/default/__cdc_meta__/capture/b6e9c580-caaa-4156-a79a-ba4f97d0e057 {"id":"b6e9c580-caaa-4156-a79a-ba4f97d0e057","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996300} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe7d2b58 b6e9c580-caaa-4156-a79a-ba4f97d0e057 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe7d2be5 1bbea387-99f4-4f89-ba6b-5cceb39a9656 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe7d2c0f acc90939-d7b6-4bb8-a061-fb4993e3fafc /tidb/cdc/default/default/changefeed/info/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"upstream-id":7365852946224692535,"namespace":"default","changefeed-id":"3d1b9799-a1cb-4947-be8e-18c4a917b77e","sink-uri":"pulsar://127.0.0.1:6650/ticdc-move-table-test-18591?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:51:52.132207376+08:00","start-ts":449575988986380290,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575993246220289} /tidb/cdc/default/default/changefeed/status/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"checkpoint-ts":449575989698101339,"min-table-barrier-ts":449575995133657091,"admin-job-type":0} /tidb/cdc/default/default/task/position/1bbea387-99f4-4f89-ba6b-5cceb39a9656/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/acc90939-d7b6-4bb8-a061-fb4993e3fafc/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b6e9c580-caaa-4156-a79a-ba4f97d0e057/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852946224692535 {"id":7365852946224692535,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/3d1b9799-a1cb-4947-be8e-18c4a917b77e {UpstreamID:7365852946224692535 Namespace:default ID:3d1b9799-a1cb-4947-be8e-18c4a917b77e SinkURI:pulsar://127.0.0.1:6650/ticdc-move-table-test-18591?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:51:52.132207376 +0800 CST StartTs:449575988986380290 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003eaa630 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575993246220289} {CheckpointTs:449575989698101339 MinTableBarrierTs:449575995133657091 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1bbea387-99f4-4f89-ba6b-5cceb39a9656 {"id":"1bbea387-99f4-4f89-ba6b-5cceb39a9656","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996314} /tidb/cdc/default/__cdc_meta__/capture/acc90939-d7b6-4bb8-a061-fb4993e3fafc {"id":"acc90939-d7b6-4bb8-a061-fb4993e3fafc","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996317} /tidb/cdc/default/__cdc_meta__/capture/b6e9c580-caaa-4156-a79a-ba4f97d0e057 {"id":"b6e9c580-caaa-4156-a79a-ba4f97d0e057","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996300} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe7d2b58 b6e9c580-caaa-4156-a79a-ba4f97d0e057 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe7d2be5 1bbea387-99f4-4f89-ba6b-5cceb39a9656 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe7d2c0f acc90939-d7b6-4bb8-a061-fb4993e3fafc /tidb/cdc/default/default/changefeed/info/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"upstream-id":7365852946224692535,"namespace":"default","changefeed-id":"3d1b9799-a1cb-4947-be8e-18c4a917b77e","sink-uri":"pulsar://127.0.0.1:6650/ticdc-move-table-test-18591?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:51:52.132207376+08:00","start-ts":449575988986380290,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575993246220289} /tidb/cdc/default/default/changefeed/status/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"checkpoint-ts":449575989698101339,"min-table-barrier-ts":449575995133657091,"admin-job-type":0} /tidb/cdc/default/default/task/position/1bbea387-99f4-4f89-ba6b-5cceb39a9656/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/acc90939-d7b6-4bb8-a061-fb4993e3fafc/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b6e9c580-caaa-4156-a79a-ba4f97d0e057/3d1b9799-a1cb-4947-be8e-18c4a917b77e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852946224692535 {"id":7365852946224692535,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x table move_table.usertable exists go: downloading github.com/IBM/sarama v1.41.2 go: downloading github.com/xdg/scram v1.0.5 go: downloading go.etcd.io/etcd/server/v3 v3.5.12 go: downloading github.com/benbjohnson/clock v1.3.5 go: downloading github.com/cakturk/go-netstat v0.0.0-20200220111822-e5b49efee7a5 go: downloading github.com/tinylib/msgp v1.1.6 go: downloading github.com/apache/pulsar-client-go v0.11.0 go: downloading github.com/aws/aws-sdk-go-v2 v1.19.1 go: downloading github.com/KimMachineGun/automemlimit v0.2.4 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/go-mysql-org/go-mysql v1.7.1-0.20240314115043-2199dfb0ba98 go: downloading gorm.io/gorm v1.24.5 go: downloading github.com/gavv/monotime v0.0.0-20190418164738-30dba4353424 go: downloading github.com/edwingeng/deque v0.0.0-20191220032131-8596380dee17 go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2 go: downloading github.com/gin-gonic/gin v1.9.1 go: downloading github.com/containerd/cgroups v1.0.4 go: downloading github.com/xdg/stringprep v1.0.3 go: downloading github.com/stretchr/objx v0.5.2 go: downloading github.com/philhofer/fwd v1.1.1 go: downloading github.com/jinzhu/now v1.1.5 go: downloading github.com/jinzhu/inflection v1.0.0 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading github.com/cilium/ebpf v0.4.0 go: downloading github.com/godbus/dbus/v5 v5.0.4 go: downloading github.com/opencontainers/runtime-spec v1.0.2 go: downloading github.com/siddontang/go v0.0.0-20180604090527-bdc77568d726 go: downloading github.com/siddontang/go-log v0.0.0-20180807004314-8d05993dda07 go: downloading github.com/gin-contrib/sse v0.1.0 go: downloading github.com/go-playground/validator/v10 v10.14.0 go: downloading github.com/ugorji/go/codec v1.2.11 go: downloading github.com/pelletier/go-toml/v2 v2.0.8 go: downloading github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 go: downloading github.com/aws/smithy-go v1.13.5 go: downloading github.com/eapache/go-resiliency v1.4.0 go: downloading github.com/eapache/queue v1.1.0 go: downloading github.com/jcmturner/gokrb5/v8 v8.4.4 go: downloading github.com/jcmturner/gofork v1.7.6 go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 go: downloading github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/linkedin/goavro/v2 v2.11.1 go: downloading github.com/pierrec/lz4 v2.6.1+incompatible go: downloading github.com/AthenZ/athenz v1.10.39 go: downloading github.com/99designs/keyring v1.2.1 go: downloading github.com/bits-and-blooms/bitset v1.4.0 go: downloading golang.org/x/mod v0.17.0 go: downloading github.com/spaolacci/murmur3 v1.1.0 go: downloading github.com/hashicorp/errwrap v1.0.0 go: downloading github.com/gabriel-vasile/mimetype v1.4.2 go: downloading github.com/leodido/go-urn v1.2.4 go: downloading github.com/go-playground/universal-translator v0.18.1 go: downloading github.com/dvsekhvalnov/jose2go v1.5.0 go: downloading github.com/mtibben/percent v0.2.1 go: downloading github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c go: downloading github.com/go-playground/locales v0.14.1 go: downloading go.etcd.io/etcd/pkg/v3 v3.5.12 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading github.com/soheilhy/cmux v0.1.5 go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0 go: downloading github.com/jonboulle/clockwork v0.4.0 go: downloading go.etcd.io/bbolt v1.3.9 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.22.0 go: downloading go.opentelemetry.io/otel/sdk v1.22.0 go: downloading go.etcd.io/etcd/raft/v3 v3.5.12 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 go: downloading github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 go: downloading github.com/golang-jwt/jwt/v4 v4.5.0 go: downloading go.etcd.io/etcd/client/v2 v2.305.12 go: downloading github.com/jcmturner/aescts/v2 v2.0.0 go: downloading github.com/hashicorp/go-uuid v1.0.3 go: downloading github.com/jcmturner/dnsutils/v2 v2.0.0 go: downloading github.com/jcmturner/rpc/v2 v2.0.3 go: downloading github.com/gorilla/websocket v1.5.1 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 go: downloading go.opentelemetry.io/proto/otlp v1.1.0 go: downloading github.com/cenkalti/backoff/v4 v4.2.1 go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 public/default start tidb cluster in /tmp/tidb_cdc_test/cli_with_auth Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public ++ date + echo '[Mon May 6 19:52:01 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:52:01 CST 2024] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.cli.13755.out cli changefeed create --start-ts=449575991383162881 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-resourcecontrol-test-14118?protocol=canal-json&enable-tidb-extension=true' =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/common_1/run.sh using Sink-Type: pulsar... <<================= go: downloading github.com/ardielle/ardielle-go v1.5.2 Create changefeed successfully! ID: 431ddcbf-d912-4920-80ea-7e115bd36b9c Info: {"upstream_id":7365852977337706292,"namespace":"default","id":"431ddcbf-d912-4920-80ea-7e115bd36b9c","sink_uri":"pulsar://127.0.0.1:6650/ticdc-resourcecontrol-test-14118?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:52:02.86085478+08:00","start_ts":449575991383162881,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575991383162881,"checkpoint_ts":449575991383162881,"checkpoint_time":"2024-05-06 19:51:45.020"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... check_changefeed_state http://127.0.0.1:2379 0f359c03-d2a8-402a-85eb-7308ad4ccb79 finished null + endpoints=http://127.0.0.1:2379 + changefeed_id=0f359c03-d2a8-402a-85eb-7308ad4ccb79 + expected_state=finished + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 0f359c03-d2a8-402a-85eb-7308ad4ccb79 -s + info='{ "upstream_id": 7365852558272560177, "namespace": "default", "id": "0f359c03-d2a8-402a-85eb-7308ad4ccb79", "state": "finished", "checkpoint_tso": 449575993007144965, "checkpoint_time": "2024-05-06 19:51:51.215", "error": null }' + echo '{ "upstream_id": 7365852558272560177, "namespace": "default", "id": "0f359c03-d2a8-402a-85eb-7308ad4ccb79", "state": "finished", "checkpoint_tso": 449575993007144965, "checkpoint_time": "2024-05-06 19:51:51.215", "error": null }' { "upstream_id": 7365852558272560177, "namespace": "default", "id": "0f359c03-d2a8-402a-85eb-7308ad4ccb79", "state": "finished", "checkpoint_tso": 449575993007144965, "checkpoint_time": "2024-05-06 19:51:51.215", "error": null } ++ echo '{' '"upstream_id":' 7365852558272560177, '"namespace":' '"default",' '"id":' '"0f359c03-d2a8-402a-85eb-7308ad4ccb79",' '"state":' '"finished",' '"checkpoint_tso":' 449575993007144965, '"checkpoint_time":' '"2024-05-06' '19:51:51.215",' '"error":' null '}' ++ jq -r .state + state=finished + [[ ! finished == \f\i\n\i\s\h\e\d ]] ++ echo '{' '"upstream_id":' 7365852558272560177, '"namespace":' '"default",' '"id":' '"0f359c03-d2a8-402a-85eb-7308ad4ccb79",' '"state":' '"finished",' '"checkpoint_tso":' 449575993007144965, '"checkpoint_time":' '"2024-05-06' '19:51:51.215",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:51:58 CST 2024] <<<<<< run test case changefeed_finish success! >>>>>> public/default ++ date + echo '[Mon May 6 19:52:04 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:52:04 CST 2024] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.cli.9499.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-common-1-test-29740?protocol=canal-json&enable-tidb-extension=true' -c=ddl-only-block-related-table + set +x [Mon May 6 19:52:04 CST 2024] <<<<<< START Pulsar consumer in resourcecontrol case >>>>>> table resourcecontrol.finish_mark not exists for 1-th check, retry later Create changefeed successfully! ID: ddl-only-block-related-table Info: {"upstream_id":7365853017852376063,"namespace":"default","id":"ddl-only-block-related-table","sink_uri":"pulsar://127.0.0.1:6650/ticdc-common-1-test-29740?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:52:04.991722096+08:00","start_ts":449575996462727170,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449575996462727170,"checkpoint_ts":449575996462727170,"checkpoint_time":"2024-05-06 19:52:04.397"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [Pipeline] // podTemplate [Pipeline] } + set +x [Mon May 6 19:52:06 CST 2024] <<<<<< START Pulsar consumer in ddl_only_block_related_table case >>>>>> [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table resourcecontrol.finish_mark exists check diff successfully table ddl_only_block_related_table.finish_mark not exists for 1-th check, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/force_replicate_table/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:52:08 CST 2024] <<<<<< run test case resourcecontrol success! >>>>>> table ddl_only_block_related_table.finish_mark not exists for 2-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa6880000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh, pid:10224, start at 2024-05-06 19:51:28.044345546 +0800 CST m=+5.421921194 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:28.052 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:28.032 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:28.032 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa6880000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh, pid:10224, start at 2024-05-06 19:51:28.044345546 +0800 CST m=+5.421921194 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:28.052 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:28.032 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:28.032 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa69340014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh, pid:10300, start at 2024-05-06 19:51:28.121862676 +0800 CST m=+5.452124468 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:28.129 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:28.126 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:28.126 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'normal'@'%' start tidb cluster failed The 2 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_only_block_related_table.finish_mark not exists for 3-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/force_replicate_table Starting Upstream PD... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... table ddl_only_block_related_table.finish_mark not exists for 4-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_only_block_related_table.finish_mark exists wait process 9053 exit for 1-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fd24a40012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb, pid:11534, start at 2024-05-06 19:52:12.869539099 +0800 CST m=+5.363290845 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:12.876 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:12.841 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:12.841 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fd24a40012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb, pid:11534, start at 2024-05-06 19:52:12.869539099 +0800 CST m=+5.363290845 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:12.876 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:12.841 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:12.841 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fd23c00008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb, pid:11600, start at 2024-05-06 19:52:12.791812128 +0800 CST m=+5.231932919 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:12.798 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:12.784 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:12.784 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cli_with_auth/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cli_with_auth/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cli_with_auth/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/cli_with_auth/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/cli_with_auth/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process 9053 exit for 2-th time... table processor_delay.t1 exists table processor_delay.t2 exists table processor_delay.t3 exists table processor_delay.t4 exists table processor_delay.t5 exists table processor_delay.t6 exists table processor_delay.t7 exists table processor_delay.t8 exists table processor_delay.t9 exists table processor_delay.t10 exists table processor_delay.t11 exists table processor_delay.t12 exists table processor_delay.t13 not exists for 1-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release wait process 9053 exit for 3-th time... table processor_delay.t13 exists table processor_delay.t14 not exists for 1-th check, retry later /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (9053) - No such process wait process 9053 exit for 4-th time... process 9053 already exit [Mon May 6 19:52:17 CST 2024] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteNotDone=return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.96479649.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiDB... + pd_host=127.0.0.1 Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.12995.out cli tso query --pd=http://127.0.0.1:2379 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t14 exists table processor_delay.t15 not exists for 1-th check, retry later + set +x + tso='449576000078741506 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576000078741506 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:52:20 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7365853017852376063 Namespace:default ID:ddl-only-block-related-table SinkURI:pulsar://127.0.0.1:6650/ticdc-common-1-test-29740?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:52:04.991722096 +0800 CST StartTs:449575996462727170 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00440cbd0 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575996607168517} {CheckpointTs:449576000290291717 MinTableBarrierTs:449576000552435716 AdminJobType:noop} span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449576000552435716, checkpointTs: 449576000290291717, state: Replicating span: {table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}, resolvedTs: 449576000552435716, checkpointTs: 449576000290291717, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449576000552435716, checkpointTs: 449576000290291717, state: Replicating span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449576000552435716, checkpointTs: 449576000290291717, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5d8d9ecb-bb31-40b1-ae48-5b528993bcb9 {"id":"5d8d9ecb-bb31-40b1-ae48-5b528993bcb9","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996337} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbec04127 5d8d9ecb-bb31-40b1-ae48-5b528993bcb9 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7365853017852376063,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"pulsar://127.0.0.1:6650/ticdc-common-1-test-29740?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:52:04.991722096+08:00","start-ts":449575996462727170,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575996607168517} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":449576000290291717,"min-table-barrier-ts":449576000552435716,"admin-job-type":0} /tidb/cdc/default/default/task/position/5d8d9ecb-bb31-40b1-ae48-5b528993bcb9/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365853017852376063 {"id":7365853017852376063,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7365853017852376063 Namespace:default ID:ddl-only-block-related-table SinkURI:pulsar://127.0.0.1:6650/ticdc-common-1-test-29740?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:52:04.991722096 +0800 CST StartTs:449575996462727170 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00440cbd0 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575996607168517} {CheckpointTs:449576000290291717 MinTableBarrierTs:449576000552435716 AdminJobType:noop} span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449576000552435716, checkpointTs: 449576000290291717, state: Replicating span: {table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}, resolvedTs: 449576000552435716, checkpointTs: 449576000290291717, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449576000552435716, checkpointTs: 449576000290291717, state: Replicating span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449576000552435716, checkpointTs: 449576000290291717, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5d8d9ecb-bb31-40b1-ae48-5b528993bcb9 {"id":"5d8d9ecb-bb31-40b1-ae48-5b528993bcb9","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996337} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbec04127 5d8d9ecb-bb31-40b1-ae48-5b528993bcb9 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7365853017852376063,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"pulsar://127.0.0.1:6650/ticdc-common-1-test-29740?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:52:04.991722096+08:00","start-ts":449575996462727170,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575996607168517} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table + grep -q 'failed to get info:' {"checkpoint-ts":449576000290291717,"min-table-barrier-ts":449576000552435716,"admin-job-type":0} /tidb/cdc/default/default/task/position/5d8d9ecb-bb31-40b1-ae48-5b528993bcb9/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365853017852376063 {"id":7365853017852376063,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7365853017852376063 Namespace:default ID:ddl-only-block-related-table SinkURI:pulsar://127.0.0.1:6650/ticdc-common-1-test-29740?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:52:04.991722096 +0800 CST StartTs:449575996462727170 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00440cbd0 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575996607168517} {CheckpointTs:449576000290291717 MinTableBarrierTs:449576000552435716 AdminJobType:noop} span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449576000552435716, checkpointTs: 449576000290291717, state: Replicating span: {table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}, resolvedTs: 449576000552435716, checkpointTs: 449576000290291717, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449576000552435716, checkpointTs: 449576000290291717, state: Replicating span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449576000552435716, checkpointTs: 449576000290291717, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5d8d9ecb-bb31-40b1-ae48-5b528993bcb9 {"id":"5d8d9ecb-bb31-40b1-ae48-5b528993bcb9","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996337} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbec04127 5d8d9ecb-bb31-40b1-ae48-5b528993bcb9 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7365853017852376063,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"pulsar://127.0.0.1:6650/ticdc-common-1-test-29740?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:52:04.991722096+08:00","start-ts":449575996462727170,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575996607168517} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":449576000290291717,"min-table-barrier-ts":449576000552435716,"admin-job-type":0} /tidb/cdc/default/default/task/position/5d8d9ecb-bb31-40b1-ae48-5b528993bcb9/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365853017852376063 {"id":7365853017852376063,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check_ts_not_forward ddl-only-block-related-table [2024/05/06 19:52:19.272 +08:00] [INFO] [main.go:61] ["table mover started"] [2024/05/06 19:52:19.274 +08:00] [INFO] [main.go:166] ["new cluster initialized"] [2024/05/06 19:52:19.275 +08:00] [DEBUG] [main.go:192] ["retrieved owner ID"] [ownerID=b6e9c580-caaa-4156-a79a-ba4f97d0e057] [2024/05/06 19:52:19.275 +08:00] [DEBUG] [main.go:199] ["retrieved owner addr"] [ownerAddr=127.0.0.1:8300] [2024/05/06 19:52:19.275 +08:00] [DEBUG] [main.go:210] ["retrieved changefeeds"] [changefeedsError="json: unsupported type: map[model.ChangeFeedID]*mvccpb.KeyValue"] [2024/05/06 19:52:19.452 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=3d1b9799-a1cb-4947-be8e-18c4a917b77e] [captureID=1bbea387-99f4-4f89-ba6b-5cceb39a9656] [processorDetail="{\"table_ids\":[]}"] [2024/05/06 19:52:19.651 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=3d1b9799-a1cb-4947-be8e-18c4a917b77e] [captureID=acc90939-d7b6-4bb8-a061-fb4993e3fafc] [processorDetail="{\"table_ids\":[]}"] [2024/05/06 19:52:19.852 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=3d1b9799-a1cb-4947-be8e-18c4a917b77e] [captureID=b6e9c580-caaa-4156-a79a-ba4f97d0e057] [processorDetail="{\"table_ids\":[106]}"] [2024/05/06 19:52:19.852 +08:00] [INFO] [main.go:75] ["task status"] [status="{\"1bbea387-99f4-4f89-ba6b-5cceb39a9656\":[],\"acc90939-d7b6-4bb8-a061-fb4993e3fafc\":[],\"b6e9c580-caaa-4156-a79a-ba4f97d0e057\":[{\"ID\":106,\"Changefeed\":\"3d1b9799-a1cb-4947-be8e-18c4a917b77e\"}]}"] [2024/05/06 19:52:19.852 +08:00] [DEBUG] [main.go:288] ["preparing HTTP API call to owner"] [formStr="cf-id=3d1b9799-a1cb-4947-be8e-18c4a917b77e&target-cp-id=acc90939-d7b6-4bb8-a061-fb4993e3fafc&table-id=106"] [2024/05/06 19:52:19.902 +08:00] [INFO] [main.go:180] ["moved table successful"] [tableID=106] [2024/05/06 19:52:19.902 +08:00] [INFO] [main.go:114] ["all tables are moved"] [sourceCapture=b6e9c580-caaa-4156-a79a-ba4f97d0e057] [targetCapture=acc90939-d7b6-4bb8-a061-fb4993e3fafc] [Mon May 6 19:52:20 CST 2024] <<<<<< START cdc server in cli_with_auth case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.1305013052.out server --log-file /tmp/tidb_cdc_test/cli_with_auth/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cli_with_auth/cdc_data --cluster-id default --config /tmp/tidb_cdc_test/cli_with_auth/server.toml + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t15 exists table processor_delay.t16 not exists for 1-th check, retry later table move_table.check1 not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/sql_mode Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t16 exists table processor_delay.t17 not exists for 1-th check, retry later table move_table.check1 not exists for 2-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:52:23 GMT < Content-Length: 872 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a35784b0-d5b5-4130-b1a4-967cb099aec5 {"id":"a35784b0-d5b5-4130-b1a4-967cb099aec5","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996341} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbf204b68 a35784b0-d5b5-4130-b1a4-967cb099aec5 /tidb/cdc/default/default/upstream/7365853128291771774 {"id":7365853128291771774,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a35784b0-d5b5-4130-b1a4-967cb099aec5 {"id":"a35784b0-d5b5-4130-b1a4-967cb099aec5","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996341} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbf204b68 a35784b0-d5b5-4130-b1a4-967cb099aec5 /tidb/cdc/default/default/upstream/7365853128291771774 {"id":7365853128291771774,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a35784b0-d5b5-4130-b1a4-967cb099aec5 {"id":"a35784b0-d5b5-4130-b1a4-967cb099aec5","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996341} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbf204b68 a35784b0-d5b5-4130-b1a4-967cb099aec5 /tidb/cdc/default/default/upstream/7365853128291771774 {"id":7365853128291771774,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:52:23 CST 2024] <<<<<< START pulsar cluster in cli_with_auth case >>>>>>' [Mon May 6 19:52:23 CST 2024] <<<<<< START pulsar cluster in cli_with_auth case >>>>>> + workdir=/tmp/tidb_cdc_test/cli_with_auth + cluster_type=normal + cd /tmp/tidb_cdc_test/cli_with_auth + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/cli_with_auth/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/cli_with_auth/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/cli_with_auth/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/cli_with_auth/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/cli_with_auth/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/cli_with_auth/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/cli_with_auth/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/cli_with_auth/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/cli_with_auth/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/cli_with_auth/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/cli_with_auth/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/cli_with_auth/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:52:23 CST 2024] <<<<<< START pulsar cluster in normal mode in cli_with_auth case >>>>>>' [Mon May 6 19:52:23 CST 2024] <<<<<< START pulsar cluster in normal mode in cli_with_auth case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/cli_with_auth/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/cli_with_auth/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/cli_with_auth/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 [Pipeline] } [Pipeline] // timeout [Pipeline] } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table processor_delay.t17 exists table processor_delay.t18 not exists for 1-th check, retry later table move_table.check1 not exists for 3-th check, retry later run task failed 1-th time, retry later + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fdc7a4000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-96v7q, pid:4620, start at 2024-05-06 19:52:23.286797145 +0800 CST m=+5.222522129 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:23.295 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:23.273 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:23.273 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fdc7a4000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-96v7q, pid:4620, start at 2024-05-06 19:52:23.286797145 +0800 CST m=+5.222522129 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:23.295 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:23.273 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:23.273 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fdc9380013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-96v7q, pid:4697, start at 2024-05-06 19:52:23.39429614 +0800 CST m=+5.284299451 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:54:23.401 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:52:23.374 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:42:23.374 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/force_replicate_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/force_replicate_table/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table processor_delay.t18 exists table processor_delay.t19 not exists for 1-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table move_table.check1 not exists for 4-th check, retry later check_ts_not_forward ddl-only-block-related-table + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public [Mon May 6 19:52:28 CST 2024] <<<<<< START cdc server in force_replicate_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.force_replicate_table.61146116.out server --log-file /tmp/tidb_cdc_test/force_replicate_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/force_replicate_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t19 exists table processor_delay.t20 not exists for 1-th check, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table move_table.check1 not exists for 5-th check, retry later Tenant not found Reason: Tenant not found + i=1 + '[' 1 -gt 20 ']' + sleep 2 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa6880000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh, pid:10224, start at 2024-05-06 19:51:28.044345546 +0800 CST m=+5.421921194 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:28.052 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:28.032 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:28.032 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa6880000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh, pid:10224, start at 2024-05-06 19:51:28.044345546 +0800 CST m=+5.421921194 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:28.052 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:28.032 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:28.032 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa69340014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh, pid:10300, start at 2024-05-06 19:51:28.121862676 +0800 CST m=+5.452124468 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:28.129 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:28.126 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:28.126 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'normal'@'%' start tidb cluster failed The 3 times to try to start tidb cluster... table processor_delay.t20 exists table processor_delay.t21 not exists for 1-th check, retry later table move_table.check1 not exists for 6-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:52:31 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e6788b91-c245-4321-b110-7df5f127c084 {"id":"e6788b91-c245-4321-b110-7df5f127c084","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996348} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbf45be31 e6788b91-c245-4321-b110-7df5f127c084 /tidb/cdc/default/default/upstream/7365853171086715195 {"id":7365853171086715195,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e6788b91-c245-4321-b110-7df5f127c084 {"id":"e6788b91-c245-4321-b110-7df5f127c084","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996348} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbf45be31 e6788b91-c245-4321-b110-7df5f127c084 /tidb/cdc/default/default/upstream/7365853171086715195 {"id":7365853171086715195,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e6788b91-c245-4321-b110-7df5f127c084 {"id":"e6788b91-c245-4321-b110-7df5f127c084","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996348} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbf45be31 e6788b91-c245-4321-b110-7df5f127c084 /tidb/cdc/default/default/upstream/7365853171086715195 {"id":7365853171086715195,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:52:31 CST 2024] <<<<<< START pulsar cluster in force_replicate_table case >>>>>>' [Mon May 6 19:52:31 CST 2024] <<<<<< START pulsar cluster in force_replicate_table case >>>>>> + workdir=/tmp/tidb_cdc_test/force_replicate_table + cluster_type=normal + cd /tmp/tidb_cdc_test/force_replicate_table + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/force_replicate_table/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/force_replicate_table/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/force_replicate_table/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/force_replicate_table/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/force_replicate_table/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/force_replicate_table/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/force_replicate_table/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/force_replicate_table/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/force_replicate_table/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/force_replicate_table/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/force_replicate_table/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/force_replicate_table/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:52:31 CST 2024] <<<<<< START pulsar cluster in normal mode in force_replicate_table case >>>>>>' [Mon May 6 19:52:31 CST 2024] <<<<<< START pulsar cluster in normal mode in force_replicate_table case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/force_replicate_table/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/force_replicate_table/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/force_replicate_table/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 + /usr/local/pulsar/bin/pulsar-admin namespaces list public table processor_delay.t21 exists table processor_delay.t22 not exists for 1-th check, retry later table move_table.check1 not exists for 7-th check, retry later + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 public/default ++ date + echo '[Mon May 6 19:52:34 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:52:34 CST 2024] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.13586.out cli changefeed create --start-ts=449576000078741506 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-cli-test-28838?protocol=canal-json&enable-tidb-extension=true' --tz=Asia/Shanghai -c=custom-changefeed-name table processor_delay.t22 exists table processor_delay.t23 not exists for 1-th check, retry later [WARN] --tz is deprecated in changefeed settings. Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7365853128291771774,"namespace":"default","id":"custom-changefeed-name","sink_uri":"pulsar://127.0.0.1:6650/ticdc-cli-test-28838?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:52:35.626664049+08:00","start_ts":449576000078741506,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576000078741506,"checkpoint_ts":449576000078741506,"checkpoint_time":"2024-05-06 19:52:18.191"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... table move_table.check1 not exists for 8-th check, retry later + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public table processor_delay.t23 exists table processor_delay.t24 not exists for 1-th check, retry later + set +x [Mon May 6 19:52:37 CST 2024] <<<<<< START Pulsar consumer in cli_with_auth case >>>>>> table test.simple not exists for 1-th check, retry later table move_table.check1 not exists for 9-th check, retry later Tenant not found Reason: Tenant not found + i=1 + '[' 1 -gt 20 ']' + sleep 2 table processor_delay.t24 exists table processor_delay.t25 not exists for 1-th check, retry later table move_table.check1 exists check diff successfully table test.simple exists table test.`simple-dash` not exists for 1-th check, retry later + /usr/local/pulsar/bin/pulsar-admin namespaces list public table processor_delay.t25 exists table processor_delay.t26 not exists for 1-th check, retry later table test.`simple-dash` exists + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7365853128291771774, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449576000616136731, "checkpoint_time": "2024-05-06 19:52:20.241", "error": null }' + echo '{ "upstream_id": 7365853128291771774, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449576000616136731, "checkpoint_time": "2024-05-06 19:52:20.241", "error": null }' { "upstream_id": 7365853128291771774, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449576000616136731, "checkpoint_time": "2024-05-06 19:52:20.241", "error": null } ++ echo '{' '"upstream_id":' 7365853128291771774, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449576000616136731, '"checkpoint_time":' '"2024-05-06' '19:52:20.241",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7365853128291771774, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449576000616136731, '"checkpoint_time":' '"2024-05-06' '19:52:20.241",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] changefeed count 1 check pass, pd_addr: http://127.0.0.1:2379 changefeed count 1 check pass, pd_addr: http://127.0.0.1:2679 public/default changefeed count 1 check pass, pd_addr: http://127.0.0.1:2779 ++ date + echo '[Mon May 6 19:52:42 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:52:42 CST 2024] <<<<<< pulsar is ready >>>>>> changefeed count 1 check pass, pd_addr: http://127.0.0.1:2379,http://127.0.0.1:2679,http://127.0.0.1:2779 table processor_delay.t26 exists table processor_delay.t27 not exists for 1-th check, retry later Create changefeed successfully! ID: e0c0e320-bf27-4c79-9020-ff9caa376c8f Info: {"upstream_id":7365853171086715195,"namespace":"default","id":"e0c0e320-bf27-4c79-9020-ff9caa376c8f","sink_uri":"pulsar://127.0.0.1:6650/ticdc-force_replicate_table-test-17190?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:52:42.984988631+08:00","start_ts":449576002760736770,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":true,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576002760736770,"checkpoint_ts":449576002760736770,"checkpoint_time":"2024-05-06 19:52:28.422"} [Mon May 6 19:52:43 CST 2024] <<<<<< START Pulsar consumer in force_replicate_table case >>>>>> Error: [CDC:ErrChangefeedUpdateRefused]changefeed update error: can only update changefeed config when it is stopped or failed update changefeed config should fail when changefeed is running, got Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config SyncPointInterval] From: To:0xc0017a99d8} {Type:update Path:[Config SyncPointRetention] From: To:0xc0017a99f0} {Type:update Path:[Config Consistent] From: To:0xc0011bf3b0} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.13989.out cli changefeed --changefeed-id custom-changefeed-name pause PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... [2024/05/06 19:52:43.830 +08:00] [INFO] [main.go:61] ["table mover started"] [2024/05/06 19:52:43.833 +08:00] [INFO] [main.go:166] ["new cluster initialized"] [2024/05/06 19:52:43.833 +08:00] [DEBUG] [main.go:192] ["retrieved owner ID"] [ownerID=b6e9c580-caaa-4156-a79a-ba4f97d0e057] [2024/05/06 19:52:43.833 +08:00] [DEBUG] [main.go:199] ["retrieved owner addr"] [ownerAddr=127.0.0.1:8300] [2024/05/06 19:52:43.834 +08:00] [DEBUG] [main.go:210] ["retrieved changefeeds"] [changefeedsError="json: unsupported type: map[model.ChangeFeedID]*mvccpb.KeyValue"] [2024/05/06 19:52:44.002 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=3d1b9799-a1cb-4947-be8e-18c4a917b77e] [captureID=1bbea387-99f4-4f89-ba6b-5cceb39a9656] [processorDetail="{\"table_ids\":[]}"] [2024/05/06 19:52:44.202 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=3d1b9799-a1cb-4947-be8e-18c4a917b77e] [captureID=acc90939-d7b6-4bb8-a061-fb4993e3fafc] [processorDetail="{\"table_ids\":[110]}"] [2024/05/06 19:52:44.402 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=3d1b9799-a1cb-4947-be8e-18c4a917b77e] [captureID=b6e9c580-caaa-4156-a79a-ba4f97d0e057] [processorDetail="{\"table_ids\":[108]}"] [2024/05/06 19:52:44.402 +08:00] [INFO] [main.go:75] ["task status"] [status="{\"1bbea387-99f4-4f89-ba6b-5cceb39a9656\":[],\"acc90939-d7b6-4bb8-a061-fb4993e3fafc\":[{\"ID\":110,\"Changefeed\":\"3d1b9799-a1cb-4947-be8e-18c4a917b77e\"}],\"b6e9c580-caaa-4156-a79a-ba4f97d0e057\":[{\"ID\":108,\"Changefeed\":\"3d1b9799-a1cb-4947-be8e-18c4a917b77e\"}]}"] [2024/05/06 19:52:44.402 +08:00] [DEBUG] [main.go:288] ["preparing HTTP API call to owner"] [formStr="cf-id=3d1b9799-a1cb-4947-be8e-18c4a917b77e&target-cp-id=b6e9c580-caaa-4156-a79a-ba4f97d0e057&table-id=110"] [2024/05/06 19:52:44.452 +08:00] [INFO] [main.go:180] ["moved table successful"] [tableID=110] [2024/05/06 19:52:44.452 +08:00] [INFO] [main.go:114] ["all tables are moved"] [sourceCapture=acc90939-d7b6-4bb8-a061-fb4993e3fafc] [targetCapture=b6e9c580-caaa-4156-a79a-ba4f97d0e057] check diff successfully table processor_delay.t27 exists table processor_delay.t28 not exists for 1-th check, retry later + set +x table move_table.check2 not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/sql_mode Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... table processor_delay.t28 exists table processor_delay.t29 not exists for 1-th check, retry later table move_table.check2 exists check diff successfully wait process cdc.test exit for 1-th time... + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=stopped + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7365853128291771774, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 449576006475055107, "checkpoint_time": "2024-05-06 19:52:42.591", "error": null }' + echo '{ "upstream_id": 7365853128291771774, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 449576006475055107, "checkpoint_time": "2024-05-06 19:52:42.591", "error": null }' { "upstream_id": 7365853128291771774, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 449576006475055107, "checkpoint_time": "2024-05-06 19:52:42.591", "error": null } ++ echo '{' '"upstream_id":' 7365853128291771774, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 449576006475055107, '"checkpoint_time":' '"2024-05-06' '19:52:42.591",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7365853128291771774, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 449576006475055107, '"checkpoint_time":' '"2024-05-06' '19:52:42.591",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.14071.out cli changefeed update --pd=http://127.0.0.1:2379,http://127.0.0.1:2679,http://127.0.0.1:2779 --config=/tmp/tidb_cdc_test/cli_with_auth/changefeed.toml --no-confirm --changefeed-id custom-changefeed-name wait process cdc.test exit for 2-th time... Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config SyncPointInterval] From: To:0xc001983468} {Type:update Path:[Config SyncPointRetention] From: To:0xc001983498} {Type:update Path:[Config Consistent] From: To:0xc001426e70} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} Update changefeed config successfully! ID: custom-changefeed-name Info: {"upstream_id":7365853128291771774,"namespace":"default","id":"custom-changefeed-name","sink_uri":"pulsar://127.0.0.1:6650/ticdc-cli-test-28838?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:52:35.626664049+08:00","start_ts":449576000078741506,"admin_job_type":1,"config":{"memory_quota":1073741824,"case_sensitive":true,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_table_monitor":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","encoder_concurrency":32,"terminator":"\r\n","enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"stopped","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":0,"checkpoint_ts":449576006475055107,"checkpoint_time":"2024-05-06 19:52:42.591"} PASS coverage: 3.0% of statements in github.com/pingcap/tiflow/... table processor_delay.t29 exists table processor_delay.t30 not exists for 1-th check, retry later table force_replicate_table.t0 exists table force_replicate_table.t1 exists table force_replicate_table.t2 exists table force_replicate_table.t3 not exists for 1-th check, retry later + set +x wait process cdc.test exit for 3-th time... + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.14104.out cli changefeed --changefeed-id custom-changefeed-name resume cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:52:50 CST 2024] <<<<<< run test case move_table success! >>>>>> table processor_delay.t30 exists table processor_delay.t31 not exists for 1-th check, retry later PASS coverage: 2.3% of statements in github.com/pingcap/tiflow/... table force_replicate_table.t3 exists table force_replicate_table.t4 not exists for 1-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + set +x Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table force_replicate_table.t4 exists table force_replicate_table.t5 not exists for 1-th check, retry later table processor_delay.t31 exists table processor_delay.t32 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa6880000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh, pid:10224, start at 2024-05-06 19:51:28.044345546 +0800 CST m=+5.421921194 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:28.052 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:28.032 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:28.032 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa6880000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh, pid:10224, start at 2024-05-06 19:51:28.044345546 +0800 CST m=+5.421921194 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:28.052 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:28.032 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:28.032 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d36fa69340014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-ndhqh, pid:10300, start at 2024-05-06 19:51:28.121862676 +0800 CST m=+5.452124468 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:53:28.129 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:51:28.126 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:41:28.126 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'normal'@'%' start tidb cluster failed + run_sql 'set global sql_mode='\''ANSI_QUOTES'\'';' 127.0.0.1 4000 + run_sql 'set global sql_mode='\''ANSI_QUOTES'\'';' 127.0.0.1 3306 ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.cli.15630.out cli tso query --pd=http://127.0.0.1:2379 table processor_delay.t32 exists table processor_delay.t33 not exists for 1-th check, retry later + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7365853128291771774, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449576009620783108, "checkpoint_time": "2024-05-06 19:52:54.591", "error": null }' + echo '{ "upstream_id": 7365853128291771774, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449576009620783108, "checkpoint_time": "2024-05-06 19:52:54.591", "error": null }' { "upstream_id": 7365853128291771774, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449576009620783108, "checkpoint_time": "2024-05-06 19:52:54.591", "error": null } ++ echo '{' '"upstream_id":' 7365853128291771774, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449576009620783108, '"checkpoint_time":' '"2024-05-06' '19:52:54.591",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7365853128291771774, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449576009620783108, '"checkpoint_time":' '"2024-05-06' '19:52:54.591",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.14191.out cli changefeed --changefeed-id custom-changefeed-name remove table force_replicate_table.t5 exists table force_replicate_table.t6 not exists for 1-th check, retry later Changefeed remove successfully. ID: custom-changefeed-name CheckpointTs: 449576009882927107 SinkURI: pulsar://127.0.0.1:6650/ticdc-cli-test-28838?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... + set +x + tso='449576009953443843 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576009953443843 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449576009953443843 + run_cdc_server --workdir /tmp/tidb_cdc_test/sql_mode --binary cdc.test [Mon May 6 19:52:57 CST 2024] <<<<<< START cdc server in sql_mode case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.1566915671.out server --log-file /tmp/tidb_cdc_test/sql_mode/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sql_mode/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t33 exists table processor_delay.t34 not exists for 1-th check, retry later table force_replicate_table.t6 exists check_data_subset force_replicate_table.t0 127.0.0.1 4000 127.0.0.1 3306 + set +x run task successfully check_data_subset force_replicate_table.t1 127.0.0.1 4000 127.0.0.1 3306 run task successfully check_data_subset force_replicate_table.t2 127.0.0.1 4000 127.0.0.1 3306 table processor_delay.t34 exists table processor_delay.t35 not exists for 1-th check, retry later run task successfully check_data_subset force_replicate_table.t3 127.0.0.1 4000 127.0.0.1 3306 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/autorandom/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:53:00 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/test-1 {UpstreamID:7365852935194214761 Namespace:default ID:test-1 SinkURI:mysql://root@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-06 19:51:40.446266317 +0800 CST StartTs:449575988869726209 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0041005a0 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575990180446211} {CheckpointTs:449575990521495558 MinTableBarrierTs:449575990521495558 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8aaf62bd-7d37-4c5b-bdb1-fadb4eb1ff09 {"id":"8aaf62bd-7d37-4c5b-bdb1-fadb4eb1ff09","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996377} /tidb/cdc/default/__cdc_meta__/capture/ed124ae1-ad47-4766-864b-52215fa9eaf5 {"id":"ed124ae1-ad47-4766-864b-52215fa9eaf5","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996297} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe6d303f ed124ae1-ad47-4766-864b-52215fa9eaf5 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbfc791cf 8aaf62bd-7d37-4c5b-bdb1-fadb4eb1ff09 /tidb/cdc/default/default/changefeed/info/test-1 {"upstream-id":7365852935194214761,"namespace":"default","changefeed-id":"test-1","sink-uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-06T19:51:40.446266317+08:00","start-ts":449575988869726209,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575990180446211} /tidb/cdc/default/default/changefeed/status/test-1 {"checkpoint-ts":449575990521495558,"min-table-barrier-ts":449575990521495558,"admin-job-type":0} /tidb/cdc/default/default/task/position/8aaf62bd-7d37-4c5b-bdb1-fadb4eb1ff09/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/ed124ae1-ad47-4766-864b-52215fa9eaf5/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852935194214761 {"id":7365852935194214761,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/test-1 {UpstreamID:7365852935194214761 Namespace:default ID:test-1 SinkURI:mysql://root@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-06 19:51:40.446266317 +0800 CST StartTs:449575988869726209 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0041005a0 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575990180446211} {CheckpointTs:449575990521495558 MinTableBarrierTs:449575990521495558 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8aaf62bd-7d37-4c5b-bdb1-fadb4eb1ff09 {"id":"8aaf62bd-7d37-4c5b-bdb1-fadb4eb1ff09","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996377} /tidb/cdc/default/__cdc_meta__/capture/ed124ae1-ad47-4766-864b-52215fa9eaf5 {"id":"ed124ae1-ad47-4766-864b-52215fa9eaf5","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996297} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe6d303f ed124ae1-ad47-4766-864b-52215fa9eaf5 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbfc791cf 8aaf62bd-7d37-4c5b-bdb1-fadb4eb1ff09 /tidb/cdc/default/default/changefeed/info/test-1 {"upstream-id":7365852935194214761,"namespace":"default","changefeed-id":"test-1","sink-uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-06T19:51:40.446266317+08:00","start-ts":449575988869726209,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575990180446211} /tidb/cdc/default/default/changefeed/status/test-1 {"checkpoint-ts":449575990521495558,"min-table-barrier-ts":449575990521495558,"admin-job-type":0} /tidb/cdc/default/default/task/position/8aaf62bd-7d37-4c5b-bdb1-fadb4eb1ff09/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/ed124ae1-ad47-4766-864b-52215fa9eaf5/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852935194214761 {"id":7365852935194214761,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/test-1 {UpstreamID:7365852935194214761 Namespace:default ID:test-1 SinkURI:mysql://root@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-05-06 19:51:40.446266317 +0800 CST StartTs:449575988869726209 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0041005a0 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575990180446211} {CheckpointTs:449575990521495558 MinTableBarrierTs:449575990521495558 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8aaf62bd-7d37-4c5b-bdb1-fadb4eb1ff09 {"id":"8aaf62bd-7d37-4c5b-bdb1-fadb4eb1ff09","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996377} /tidb/cdc/default/__cdc_meta__/capture/ed124ae1-ad47-4766-864b-52215fa9eaf5 {"id":"ed124ae1-ad47-4766-864b-52215fa9eaf5","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996297} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbe6d303f ed124ae1-ad47-4766-864b-52215fa9eaf5 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbfc791cf 8aaf62bd-7d37-4c5b-bdb1-fadb4eb1ff09 /tidb/cdc/default/default/changefeed/info/test-1 {"upstream-id":7365852935194214761,"namespace":"default","changefeed-id":"test-1","sink-uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-05-06T19:51:40.446266317+08:00","start-ts":449575988869726209,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575990180446211} /tidb/cdc/default/default/changefeed/status/test-1 {"checkpoint-ts":449575990521495558,"min-table-barrier-ts":449575990521495558,"admin-job-type":0} /tidb/cdc/default/default/task/position/8aaf62bd-7d37-4c5b-bdb1-fadb4eb1ff09/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/ed124ae1-ad47-4766-864b-52215fa9eaf5/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"+ grep -q 'etcd info' error":null,"warning":null} /tidb/cdc/default/default/upstream/7365852935194214761 {"id":7365852935194214761,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449576009953443843 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-2 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.cli.15731.out cli changefeed create --start-ts=449576009953443843 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-2 run task successfully check_data_subset force_replicate_table.t4 127.0.0.1 4000 127.0.0.1 3306 changefeed count 0 check pass, pd_addr: http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.14268.out cli changefeed create '--sink-uri=pulsar://127.0.0.1:6650/ticdc-cli-test-28838?protocol=canal-json&enable-tidb-extension=true' --tz=Asia/Shanghai -c=custom-changefeed-name run task successfully check_data_subset force_replicate_table.t5 127.0.0.1 4000 127.0.0.1 3306 [WARN] --tz is deprecated in changefeed settings. Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7365853128291771774,"namespace":"default","id":"custom-changefeed-name","sink_uri":"pulsar://127.0.0.1:6650/ticdc-cli-test-28838?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:53:01.752500824+08:00","start_ts":449576011468898305,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576011468898305,"checkpoint_ts":449576011468898305,"checkpoint_time":"2024-05-06 19:53:01.641"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... table processor_delay.t35 exists table processor_delay.t36 not exists for 1-th check, retry later run task successfully check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 id=19,a=NULL doesn't exist in downstream table force_replicate_table.t6 run task failed 1-th time, retry later + set +x Create changefeed successfully! ID: test-2 Info: {"upstream_id":7365852935194214761,"namespace":"default","id":"test-2","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-06T19:53:03.318448934+08:00","start_ts":449576009953443843,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576009953443843,"checkpoint_ts":449576009953443843,"checkpoint_time":"2024-05-06 19:52:55.860"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 id=7,a=NULL doesn't exist in downstream table force_replicate_table.t6 run task failed 2-th time, retry later + set +x + run_sql 'use test; create table t2(id bigint primary key, a date); insert into t2 values(1, '\''2023-02-08'\'');' 127.0.0.1 4000 table processor_delay.t36 exists table processor_delay.t37 exists table processor_delay.t38 not exists for 1-th check, retry later + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7365853128291771774, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449576012504367109, "checkpoint_time": "2024-05-06 19:53:05.591", "error": null }' + echo '{ "upstream_id": 7365853128291771774, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449576012504367109, "checkpoint_time": "2024-05-06 19:53:05.591", "error": null }' { "upstream_id": 7365853128291771774, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449576012504367109, "checkpoint_time": "2024-05-06 19:53:05.591", "error": null } ++ echo '{' '"upstream_id":' 7365853128291771774, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449576012504367109, '"checkpoint_time":' '"2024-05-06' '19:53:05.591",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7365853128291771774, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449576012504367109, '"checkpoint_time":' '"2024-05-06' '19:53:05.591",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] table processor_delay.t38 exists table processor_delay.t39 not exists for 1-th check, retry later + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.14372.out cli unsafe delete-service-gc-safepoint Confirm that you know what this command will do and use it at your own risk [Y/N] check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 CDC service GC safepoint truncated in PD! PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... start tidb cluster in /tmp/tidb_cdc_test/autorandom Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... table processor_delay.t39 exists table processor_delay.t40 not exists for 1-th check, retry later run task successfully wait process cdc.test exit for 1-th time... + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.14405.out cli unsafe reset --no-confirm --pd=http://127.0.0.1:2379 wait process cdc.test exit for 2-th time... reset and all metadata truncated in PD! PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... wait process cdc.test exit for 3-th time... table processor_delay.t40 exists table processor_delay.t41 not exists for 1-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:53:11 CST 2024] <<<<<< run test case force_replicate_table success! >>>>>> + set +x table processor_delay.t41 exists table processor_delay.t42 not exists for 1-th check, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t42 exists table processor_delay.t43 not exists for 1-th check, retry later + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.14510.out cli unsafe resolve-lock --region=56 PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.14542.out cli unsafe resolve-lock --region=56 --ts=449576014116552708 table processor_delay.t43 exists table processor_delay.t44 not exists for 1-th check, retry later PASS coverage: 2.2% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 218 100 212 100 6 145k 4210 --:--:-- --:--:-- --:--:-- 207k { "error_msg": "[CDC:ErrAPIInvalidParam]invalid log level: json: cannot unmarshal string into Go value of type struct { Level string \"json:\\\"log_level\\\"\" }", "error_code": "CDC:ErrAPIInvalidParam" table processor_delay.t44 exists table processor_delay.t45 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t45 exists table processor_delay.t46 not exists for 1-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc already exists) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370121ac0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-v1jxz, pid:31297, start at 2024-05-06 19:53:18.228117495 +0800 CST m=+5.237040375 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:18.234 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:18.237 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:18.237 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370121ac0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-v1jxz, pid:31297, start at 2024-05-06 19:53:18.228117495 +0800 CST m=+5.237040375 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:18.234 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:18.237 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:18.237 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370123540014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-v1jxz, pid:31372, start at 2024-05-06 19:53:18.31893746 +0800 CST m=+5.267380825 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:18.328 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:18.293 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:18.293 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } Logging trace to /tmp/tidb_cdc_test/autorandom/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/autorandom/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } } % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 182 100 182 0 0 210k 0 --:--:-- --:--:-- --:--:-- 177k { "version": "v8.1.0-alpha-23-ge1a27adc6", "git_hash": "e1a27adc69acc9d50db0c3fea2945bbd6d983e30", "id": "ea822dfa-7dad-4dec-927b-6ab81835bc09", "pid": 13055, "is_owner": true [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } }wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... table processor_delay.t46 exists table processor_delay.t47 not exists for 1-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Mon May 6 19:53:23 CST 2024] <<<<<< run test case cli_with_auth success! >>>>>> [Mon May 6 19:53:23 CST 2024] <<<<<< START cdc server in autorandom case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.autorandom.3278132783.out server --log-file /tmp/tidb_cdc_test/autorandom/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/autorandom/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t47 exists table processor_delay.t48 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:53:26 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4b180884-7b3e-4f99-88e8-dccd2e8265d7 {"id":"4b180884-7b3e-4f99-88e8-dccd2e8265d7","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996403} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc0221f31 4b180884-7b3e-4f99-88e8-dccd2e8265d7 /tidb/cdc/default/default/upstream/7365853404995714112 {"id":7365853404995714112,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4b180884-7b3e-4f99-88e8-dccd2e8265d7 {"id":"4b180884-7b3e-4f99-88e8-dccd2e8265d7","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996403} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc0221f31 4b180884-7b3e-4f99-88e8-dccd2e8265d7 /tidb/cdc/default/default/upstream/7365853404995714112 {"id":7365853404995714112,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4b180884-7b3e-4f99-88e8-dccd2e8265d7 {"id":"4b180884-7b3e-4f99-88e8-dccd2e8265d7","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996403} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc0221f31 4b180884-7b3e-4f99-88e8-dccd2e8265d7 /tidb/cdc/default/default/upstream/7365853404995714112 {"id":7365853404995714112,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:53:26 CST 2024] <<<<<< START pulsar cluster in autorandom case >>>>>>' [Mon May 6 19:53:26 CST 2024] <<<<<< START pulsar cluster in autorandom case >>>>>> + workdir=/tmp/tidb_cdc_test/autorandom + cluster_type=normal + cd /tmp/tidb_cdc_test/autorandom + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/autorandom/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/autorandom/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/autorandom/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/autorandom/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/autorandom/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/autorandom/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/autorandom/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/autorandom/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/autorandom/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/autorandom/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/autorandom/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/autorandom/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:53:26 CST 2024] <<<<<< START pulsar cluster in normal mode in autorandom case >>>>>>' [Mon May 6 19:53:26 CST 2024] <<<<<< START pulsar cluster in normal mode in autorandom case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/autorandom/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/autorandom/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/autorandom/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 table processor_delay.t48 exists table processor_delay.t49 not exists for 1-th check, retry later + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 table processor_delay.t49 exists table processor_delay.t50 not exists for 1-th check, retry later + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public table processor_delay.t50 exists check diff failed 1-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/savepoint/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... check diff successfully wait process cdc.test exit for 1-th time... pass check, checkpoint tso not forward after 10s run task successfully wait process 9652 exit for 1-th time... wait process 9652 exit for 2-th time... wait process 9652 exit for 3-th time... wait process 9652 exit for 4-th time... wait process 9652 exit for 5-th time... /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (9652) - No such process wait process 9652 exit for 6-th time... process 9652 already exit [Mon May 6 19:53:29 CST 2024] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.1040610408.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:53:32 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7365853017852376063 Namespace:default ID:ddl-only-block-related-table SinkURI:pulsar://127.0.0.1:6650/ticdc-common-1-test-29740?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:52:04.991722096 +0800 CST StartTs:449575996462727170 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003cc02d0 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575996607168517} {CheckpointTs:449576000657293322 MinTableBarrierTs:449576019439910922 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449576000657293322, checkpointTs: 449576000657293322, state: Preparing span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449576000657293322, checkpointTs: 449576000657293322, state: Preparing span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449576000657293322, checkpointTs: 449576000657293322, state: Preparing span: {table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}, resolvedTs: 449576000657293322, checkpointTs: 449576000657293322, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/219e68a7-774f-49e1-b0a4-fa62542c3fb2 {"id":"219e68a7-774f-49e1-b0a4-fa62542c3fb2","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996410} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbec0433c 219e68a7-774f-49e1-b0a4-fa62542c3fb2 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7365853017852376063,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"pulsar://127.0.0.1:6650/ticdc-common-1-test-29740?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:52:04.991722096+08:00","start-ts":449575996462727170,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575996607168517} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":449576000657293322,"min-table-barrier-ts":449576019439910922,"admin-job-type":0} /tidb/cdc/default/default/task/position/219e68a7-774f-49e1-b0a4-fa62542c3fb2/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365853017852376063 {"id":7365853017852376063,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7365853017852376063 Namespace:default ID:ddl-only-block-related-table SinkURI:pulsar://127.0.0.1:6650/ticdc-common-1-test-29740?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:52:04.991722096 +0800 CST StartTs:449575996462727170 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003cc02d0 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575996607168517} {CheckpointTs:449576000657293322 MinTableBarrierTs:449576019439910922 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449576000657293322, checkpointTs: 449576000657293322, state: Preparing span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449576000657293322, checkpointTs: 449576000657293322, state: Preparing span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449576000657293322, checkpointTs: 449576000657293322, state: Preparing span: {table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}, resolvedTs: 449576000657293322, checkpointTs: 449576000657293322, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/219e68a7-774f-49e1-b0a4-fa62542c3fb2 {"id":"219e68a7-774f-49e1-b0a4-fa62542c3fb2","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996410} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbec0433c 219e68a7-774f-49e1-b0a4-fa62542c3fb2 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7365853017852376063,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"pulsar://127.0.0.1:6650/ticdc-common-1-test-29740?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:52:04.991722096+08:00","start-ts":449575996462727170,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575996607168517} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":449576000657293322,"min-table-barrier-ts":449576019439910922,"admin-job-type":0} /tidb/cdc/default/default/task/position/219e68a7-774f-49e1-b0a4-fa62542c3fb2/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365853017852376063 {"id":7365853017852376063,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7365853017852376063 Namespace:default ID:ddl-only-block-related-table SinkURI:pulsar://127.0.0.1:6650/ticdc-common-1-test-29740?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-05-06 19:52:04.991722096 +0800 CST StartTs:449575996462727170 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003cc02d0 State:normal Error: Warning: CreatorVersion:v8.1.0-alpha-23-ge1a27adc6 Epoch:449575996607168517} {CheckpointTs:449576000657293322 MinTableBarrierTs:449576019439910922 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449576000657293322, checkpointTs: 449576000657293322, state: Preparing span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449576000657293322, checkpointTs: 449576000657293322, state: Preparing span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449576000657293322, checkpointTs: 449576000657293322, state: Preparing span: {table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}, resolvedTs: 449576000657293322, checkpointTs: 449576000657293322, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/219e68a7-774f-49e1-b0a4-fa62542c3fb2 {"id":"219e68a7-774f-49e1-b0a4-fa62542c3fb2","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996410} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dbec0433c 219e68a7-774f-49e1-b0a4-fa62542c3fb2 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7365853017852376063,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"pulsar://127.0.0.1:6650/ticdc-common-1-test-29740?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-05-06T19:52:04.991722096+08:00","start-ts":449575996462727170,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"pulsar-config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"debezium-disable-schema":false,"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefee+ grep -q 'etcd info' d-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v8.1.0-alpha-23-ge1a27adc6","epoch":449575996607168517} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":449576000657293322,"min-table-barrier-ts":449576019439910922,"admin-job-type":0} /tidb/cdc/default/default/task/position/219e68a7-774f-49e1-b0a4-fa62542c3fb2/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7365853017852376063 {"id":7365853017852376063,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x check diff failed 1-th time, retry later check diff failed 2-th time, retry later wait process cdc.test exit for 2-th time... public/default ++ date + echo '[Mon May 6 19:53:35 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:53:35 CST 2024] <<<<<< pulsar is ready >>>>>> wait process cdc.test exit for 3-th time... Create changefeed successfully! ID: ed8a921c-0b9f-4357-8c76-0fc8825ff40a Info: {"upstream_id":7365853404995714112,"namespace":"default","id":"ed8a921c-0b9f-4357-8c76-0fc8825ff40a","sink_uri":"pulsar://127.0.0.1:6650/ticdc-autorandom-test-12607?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:53:36.153553939+08:00","start_ts":449576020328316932,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576020328316932,"checkpoint_ts":449576020328316932,"checkpoint_time":"2024-05-06 19:53:35.437"} [Mon May 6 19:53:36 CST 2024] <<<<<< START Pulsar consumer in autorandom case >>>>>> start tidb cluster in /tmp/tidb_cdc_test/savepoint Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:53:36 CST 2024] <<<<<< run test case processor_etcd_worker_delay success! >>>>>> check diff successfully check_ts_forward ddl-only-block-related-table table autorandom_test.table_a not exists for 1-th check, retry later table autorandom_test.table_a exists check diff failed 1-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release changefeed is working normally rts: 449576020750630900->449576021012774900 checkpoint: 449576020750630900->449576021012774900 run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 4-th time... check diff successfully cdc.test: no process found wait process cdc.test exit for 5-th time... process cdc.test already exit [Mon May 6 19:53:41 CST 2024] <<<<<< run test case ddl_only_block_related_table success! >>>>>> wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:53:43 CST 2024] <<<<<< run test case autorandom success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3702dbe0000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb, pid:15618, start at 2024-05-06 19:53:46.501756837 +0800 CST m=+5.323181066 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:46.509 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:46.488 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:46.488 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3702dbe0000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb, pid:15618, start at 2024-05-06 19:53:46.501756837 +0800 CST m=+5.323181066 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:46.509 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:46.488 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:46.488 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3702dcb8000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb, pid:15702, start at 2024-05-06 19:53:46.561637518 +0800 CST m=+5.328637643 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:55:46.568 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:53:46.542 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:43:46.542 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/savepoint/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/savepoint/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/sink_hang/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.cli.16961.out cli tso query --pd=http://127.0.0.1:2379 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/changefeed_auto_stop/run.sh using Sink-Type: pulsar... <<================= The 1 times to try to start tidb cluster... + set +x + tso='449576024627740673 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576024627740673 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Mon May 6 19:53:53 CST 2024] <<<<<< START cdc server in savepoint case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.1700017002.out server --log-file /tmp/tidb_cdc_test/savepoint/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/savepoint/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/generate_column/run.sh using Sink-Type: pulsar... <<================= start tidb cluster in /tmp/tidb_cdc_test/sink_hang Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/changefeed_auto_stop Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... Verifying downstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:53:56 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/341a97e1-7be4-4434-a60c-f8fae443ab07 {"id":"341a97e1-7be4-4434-a60c-f8fae443ab07","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996433} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc08e5437 341a97e1-7be4-4434-a60c-f8fae443ab07 /tidb/cdc/default/default/upstream/7365853530640670982 {"id":7365853530640670982,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/341a97e1-7be4-4434-a60c-f8fae443ab07 {"id":"341a97e1-7be4-4434-a60c-f8fae443ab07","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996433} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc08e5437 341a97e1-7be4-4434-a60c-f8fae443ab07 /tidb/cdc/default/default/upstream/7365853530640670982 {"id":7365853530640670982,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/341a97e1-7be4-4434-a60c-f8fae443ab07 {"id":"341a97e1-7be4-4434-a60c-f8fae443ab07","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996433} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc08e5437 341a97e1-7be4-4434-a60c-f8fae443ab07 /tidb/cdc/default/default/upstream/7365853530640670982 {"id":7365853530640670982,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:53:56 CST 2024] <<<<<< START pulsar cluster in savepoint case >>>>>>' [Mon May 6 19:53:56 CST 2024] <<<<<< START pulsar cluster in savepoint case >>>>>> + workdir=/tmp/tidb_cdc_test/savepoint + cluster_type=normal + cd /tmp/tidb_cdc_test/savepoint + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/savepoint/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/savepoint/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/savepoint/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/savepoint/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/savepoint/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/savepoint/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/savepoint/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/savepoint/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/savepoint/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/savepoint/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/savepoint/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/savepoint/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:53:56 CST 2024] <<<<<< START pulsar cluster in normal mode in savepoint case >>>>>>' [Mon May 6 19:53:56 CST 2024] <<<<<< START pulsar cluster in normal mode in savepoint case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/savepoint/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/savepoint/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/savepoint/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + '[' pulsar == mysql ']' + stop_tidb_cluster ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2722 0 --:--:-- --:--:-- --:--:-- 2728 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-06 19:53:49.187","puller_resolved_ts":"2024-05-06 19:53:42.187","last_synced_ts":"2024-05-06 19:51:33.487","now_ts":"2024-05-06 19:53:50.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '19:53:49.187","puller_resolved_ts":"2024-05-06' '19:53:42.187","last_synced_ts":"2024-05-06' '19:51:33.487","now_ts":"2024-05-06' '19:53:50.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' + kill_pd ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo + info='jenkins 11003 7.7 0.0 13717980 143156 ? Sl 19:51 0:12 pd-server --advertise-client-urls http://127.0.0.1:2379 --client-urls http://0.0.0.0:2379 --advertise-peer-urls http://127.0.0.1:2380 --peer-urls http://0.0.0.0:2380 --config /tmp/tidb_cdc_test/synced_status_with_redo/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status_with_redo/pd1.log --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/pd1 --name=pd1 --initial-cluster=pd1=http://127.0.0.1:2380 jenkins 11062 5.2 0.0 13521820 138216 ? Sl 19:51 0:08 pd-server --advertise-client-urls http://127.0.0.1:2479 --client-urls http://0.0.0.0:2479 --advertise-peer-urls http://127.0.0.1:2480 --peer-urls http://0.0.0.0:2480 --config /tmp/tidb_cdc_test/synced_status_with_redo/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status_with_redo/down_pd.log --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/down_pd' ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo ++ awk '{print $2}' ++ xargs kill -9 + sleep 20 {"level":"warn","ts":1714996438.2571704,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002f83880/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714996438.257236,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1714996438.2691529,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002cfca80/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1714996438.2691994,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1714996438.2708762,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00220ce00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714996438.2709334,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} Tenant not found Reason: Tenant not found + i=1 + '[' 1 -gt 20 ']' + sleep 2 {"level":"warn","ts":"2024-05-06T19:54:03.096284+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000aad6c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:54:03.096485+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0010c2000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:54:03.156496+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0012ec000/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + /usr/local/pulsar/bin/pulsar-admin namespaces list public + stop_tidb_cluster ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370408c00011 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-807c6, pid:8901, start at 2024-05-06 19:54:05.762423983 +0800 CST m=+5.295084072 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:56:05.770 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:54:05.744 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:44:05.744 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) public/default ++ date + echo '[Mon May 6 19:54:07 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:54:07 CST 2024] <<<<<< pulsar is ready >>>>>> + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.cli.17530.out cli changefeed create --start-ts=449576024627740673 '--sink-uri=pulsar://127.0.0.1:6650/ticdc-savepoint-test-9177?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: 378c300f-d182-4693-8c5c-af26e36c5cae Info: {"upstream_id":7365853530640670982,"namespace":"default","id":"378c300f-d182-4693-8c5c-af26e36c5cae","sink_uri":"pulsar://127.0.0.1:6650/ticdc-savepoint-test-9177?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-05-06T19:54:08.327095285+08:00","start_ts":449576024627740673,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"pulsar_config":{"connection-timeout":5,"operation-timeout":30,"batching-max-messages":1000,"batching-max-publish-delay":10,"send-timeout":30},"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576024627740673,"checkpoint_ts":449576024627740673,"checkpoint_time":"2024-05-06 19:53:51.838"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370407140014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-55h44, pid:11650, start at 2024-05-06 19:54:05.674689577 +0800 CST m=+5.543511605 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:56:05.682 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:54:05.686 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:44:05.686 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370407140014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-55h44, pid:11650, start at 2024-05-06 19:54:05.674689577 +0800 CST m=+5.543511605 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:56:05.682 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:54:05.686 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:44:05.686 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370406fc0015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-55h44, pid:11716, start at 2024-05-06 19:54:05.67317608 +0800 CST m=+5.492121409 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:56:05.681 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:54:05.681 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:44:05.681 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370408c00011 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-807c6, pid:8901, start at 2024-05-06 19:54:05.762423983 +0800 CST m=+5.295084072 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:56:05.770 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:54:05.744 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:44:05.744 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d37040af80015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-807c6, pid:8982, start at 2024-05-06 19:54:05.920430006 +0800 CST m=+5.397019134 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:56:05.927 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:54:05.886 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:44:05.886 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sink_hang/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sink_hang/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } {"level":"warn","ts":"2024-05-06T19:54:09.096808+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000aad6c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:54:09.097237+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0010c2000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:54:09.157584+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0012ec000/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} + set +x [Mon May 6 19:54:09 CST 2024] <<<<<< START Pulsar consumer in savepoint case >>>>>> table savepoint.finish_mark not exists for 1-th check, retry later ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed [Mon May 6 19:54:10 CST 2024] <<<<<< START cdc server in sink_hang case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkExecDMLError=2*return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_hang.1026510267.out server --log-file /tmp/tidb_cdc_test/sink_hang/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sink_hang/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ***************** properties ***************** "workload"="core" "updateproportion"="0" "insertproportion"="0" "mysql.port"="4000" "readallfields"="true" "scanproportion"="0" "requestdistribution"="uniform" "dotransactions"="false" "operationcount"="0" "mysql.host"="127.0.0.1" "mysql.db"="changefeed_auto_stop_1" "readproportion"="0" "recordcount"="20" "mysql.user"="root" "threadcount"="4" ********************************************** Run finished, takes 13.165549ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2845.4, Avg(us): 2453, Min(us): 1039, Max(us): 6143, 95th(us): 7000, 99th(us): 7000 table savepoint.finish_mark not exists for 2-th check, retry later Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } ***************** properties ***************** "readallfields"="true" "updateproportion"="0" "operationcount"="0" "insertproportion"="0" "workload"="core" "mysql.db"="changefeed_auto_stop_2" "scanproportion"="0" "mysql.host"="127.0.0.1" "recordcount"="20" "readproportion"="0" "requestdistribution"="uniform" "mysql.port"="4000" "mysql.user"="root" "dotransactions"="false" "threadcount"="4" ********************************************** Run finished, takes 11.484363ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3093.5, Avg(us): 2141, Min(us): 1186, Max(us): 5008, 95th(us): 5000, 99th(us): 6000 [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:54:13 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1e2384a9-95a1-4bca-b40a-385881c0a26f {"id":"1e2384a9-95a1-4bca-b40a-385881c0a26f","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996451} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc0d5a138 1e2384a9-95a1-4bca-b40a-385881c0a26f /tidb/cdc/default/default/upstream/7365853603633247668 {"id":7365853603633247668,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1e2384a9-95a1-4bca-b40a-385881c0a26f {"id":"1e2384a9-95a1-4bca-b40a-385881c0a26f","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996451} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc0d5a138 1e2384a9-95a1-4bca-b40a-385881c0a26f /tidb/cdc/default/default/upstream/7365853603633247668 {"id":7365853603633247668,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1e2384a9-95a1-4bca-b40a-385881c0a26f {"id":"1e2384a9-95a1-4bca-b40a-385881c0a26f","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996451} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc0d5a138 1e2384a9-95a1-4bca-b40a-385881c0a26f /tidb/cdc/default/default/upstream/7365853603633247668 {"id":7365853603633247668,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Mon May 6 19:54:14 CST 2024] <<<<<< START Pulsar consumer in sink_hang case >>>>>> table savepoint.finish_mark exists ***************** properties ***************** "scanproportion"="0" "updateproportion"="0" "operationcount"="0" "insertproportion"="0" "readallfields"="true" "mysql.user"="root" "recordcount"="20" "mysql.db"="changefeed_auto_stop_3" "workload"="core" "readproportion"="0" "mysql.port"="4000" "dotransactions"="false" "mysql.host"="127.0.0.1" "requestdistribution"="uniform" "threadcount"="4" ********************************************** Run finished, takes 14.186042ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2863.2, Avg(us): 2672, Min(us): 1191, Max(us): 7462, 95th(us): 8000, 99th(us): 8000 check diff successfully check_changefeed_status 127.0.0.1:8300 69791146-eb39-418a-b425-7a4974e027fe normal last_error null + endpoint=127.0.0.1:8300 + changefeed_id=69791146-eb39-418a-b425-7a4974e027fe + expected_state=normal + field=last_error + error_pattern=null ++ curl 127.0.0.1:8300/api/v2/changefeeds/69791146-eb39-418a-b425-7a4974e027fe/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 86 100 86 0 0 628 0 --:--:-- --:--:-- --:--:-- 627 100 86 100 86 0 0 627 0 --:--:-- --:--:-- --:--:-- 627 + info='{"state":"normal","resolved_ts":449576030422695940,"checkpoint_ts":449576030422695940}' + echo '{"state":"normal","resolved_ts":449576030422695940,"checkpoint_ts":449576030422695940}' {"state":"normal","resolved_ts":449576030422695940,"checkpoint_ts":449576030422695940} ++ echo '{"state":"normal","resolved_ts":449576030422695940,"checkpoint_ts":449576030422695940}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] + [[ -z last_error ]] ++ echo '{"state":"normal","resolved_ts":449576030422695940,"checkpoint_ts":449576030422695940}' ++ jq -r .last_error.message + error_msg=null + [[ ! null =~ null ]] run task successfully check_changefeed_status 127.0.0.1:8300 69791146-eb39-418a-b425-7a4974e027fe normal last_warning null + endpoint=127.0.0.1:8300 + changefeed_id=69791146-eb39-418a-b425-7a4974e027fe + expected_state=normal + field=last_warning + error_pattern=null ++ curl 127.0.0.1:8300/api/v2/changefeeds/69791146-eb39-418a-b425-7a4974e027fe/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 86 100 86 0 0 671 0 --:--:-- --:--:-- --:--:-- 677 + info='{"state":"normal","resolved_ts":449576030422695940,"checkpoint_ts":449576030422695940}' + echo '{"state":"normal","resolved_ts":449576030422695940,"checkpoint_ts":449576030422695940}' {"state":"normal","resolved_ts":449576030422695940,"checkpoint_ts":449576030422695940} ++ echo '{"state":"normal","resolved_ts":449576030422695940,"checkpoint_ts":449576030422695940}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] + [[ -z last_warning ]] ++ echo '{"state":"normal","resolved_ts":449576030422695940,"checkpoint_ts":449576030422695940}' ++ jq -r .last_warning.message + error_msg=null + [[ ! null =~ null ]] run task successfully table sink_hang.t1 not exists for 1-th check, retry later wait process cdc.test exit for 1-th time... ***************** properties ***************** "insertproportion"="0" "mysql.port"="4000" "mysql.db"="changefeed_auto_stop_4" "threadcount"="4" "mysql.host"="127.0.0.1" "workload"="core" "recordcount"="20" "requestdistribution"="uniform" "mysql.user"="root" "operationcount"="0" "dotransactions"="false" "scanproportion"="0" "readallfields"="true" "updateproportion"="0" "readproportion"="0" ********************************************** Run finished, takes 10.683611ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3337.2, Avg(us): 1992, Min(us): 1069, Max(us): 4935, 95th(us): 5000, 99th(us): 5000 [Mon May 6 19:54:15 CST 2024] <<<<<< START cdc server in changefeed_auto_stop case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_auto_stop.1325013252.out server --log-file /tmp/tidb_cdc_test/changefeed_auto_stop/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_auto_stop/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0{"level":"warn","ts":"2024-05-06T19:54:15.098254+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0010c2000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:54:15.098419+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000aad6c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:54:15.159257+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0012ec000/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... process cdc.test already exit [Mon May 6 19:54:15 CST 2024] <<<<<< run test case savepoint success! >>>>>> table sink_hang.t1 exists table sink_hang.t2 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:54:18 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dc201d6b-b4ea-461e-906e-388aa881ffdf {"id":"dc201d6b-b4ea-461e-906e-388aa881ffdf","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996455} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc0da4a9a dc201d6b-b4ea-461e-906e-388aa881ffdf /tidb/cdc/default/default/upstream/7365853613935670194 {"id":7365853613935670194,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dc201d6b-b4ea-461e-906e-388aa881ffdf {"id":"dc201d6b-b4ea-461e-906e-388aa881ffdf","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996455} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc0da4a9a dc201d6b-b4ea-461e-906e-388aa881ffdf /tidb/cdc/default/default/upstream/7365853613935670194 {"id":7365853613935670194,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dc201d6b-b4ea-461e-906e-388aa881ffdf {"id":"dc201d6b-b4ea-461e-906e-388aa881ffdf","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996455} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc0da4a9a dc201d6b-b4ea-461e-906e-388aa881ffdf /tidb/cdc/default/default/upstream/7365853613935670194 {"id":7365853613935670194,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Mon May 6 19:54:18 CST 2024] <<<<<< START cdc server in changefeed_auto_stop case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/pipeline/ProcessorSyncResolvedError=1*return(true);github.com/pingcap/tiflow/cdc/processor/ProcessorUpdatePositionDelaying=sleep(1000)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_auto_stop.1329913301.out server --log-file /tmp/tidb_cdc_test/changefeed_auto_stop/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_auto_stop/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table sink_hang.t2 exists check diff failed 1-th time, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0{"level":"warn","ts":"2024-05-06T19:54:21.099416+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0010c2000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:54:21.100046+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000aad6c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:54:21.160483+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0012ec000/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} check diff failed 2-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:54:21 GMT < Content-Length: 1297 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1afe0793-1a6e-41e0-bc8c-820263de968b {"id":"1afe0793-1a6e-41e0-bc8c-820263de968b","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996458} /tidb/cdc/default/__cdc_meta__/capture/dc201d6b-b4ea-461e-906e-388aa881ffdf {"id":"dc201d6b-b4ea-461e-906e-388aa881ffdf","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996455} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc0da4a9a dc201d6b-b4ea-461e-906e-388aa881ffdf /tidb/cdc/default/__cdc_meta__/owner/22318f4dc0da4ad0 1afe0793-1a6e-41e0-bc8c-820263de968b /tidb/cdc/default/default/upstream/7365853613935670194 {"id":7365853613935670194,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1afe0793-1a6e-41e0-bc8c-820263de968b {"id":"1afe0793-1a6e-41e0-bc8c-820263de968b","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996458} /tidb/cdc/default/__cdc_meta__/capture/dc201d6b-b4ea-461e-906e-388aa881ffdf {"id":"dc201d6b-b4ea-461e-906e-388aa881ffdf","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996455} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc0da4a9a dc201d6b-b4ea-461e-906e-388aa881ffdf /tidb/cdc/default/__cdc_meta__/owner/22318f4dc0da4ad0 1afe0793-1a6e-41e0-bc8c-820263de968b /tidb/cdc/default/default/upstream/7365853613935670194 {"id":7365853613935670194,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1afe0793-1a6e-41e0-bc8c-820263de968b {"id":"1afe0793-1a6e-41e0-bc8c-820263de968b","address":"127.0.0.1:8302","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996458} /tidb/cdc/default/__cdc_meta__/capture/dc201d6b-b4ea-461e-906e-388aa881ffdf {"id":"dc201d6b-b4ea-461e-906e-388aa881ffdf","address":"127.0.0.1:8301","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996455} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc0da4a9a dc201d6b-b4ea-461e-906e-388aa881ffdf /tidb/cdc/default/__cdc_meta__/owner/22318f4dc0da4ad0 1afe0793-1a6e-41e0-bc8c-820263de968b /tidb/cdc/default/default/upstream/7365853613935670194 {"id":7365853613935670194,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ++ date + echo '[Mon May 6 19:54:21 CST 2024] <<<<<< START pulsar cluster in changefeed_auto_stop case >>>>>>' [Mon May 6 19:54:21 CST 2024] <<<<<< START pulsar cluster in changefeed_auto_stop case >>>>>> + workdir=/tmp/tidb_cdc_test/changefeed_auto_stop + cluster_type=normal + cd /tmp/tidb_cdc_test/changefeed_auto_stop + DEFAULT_PULSAR_HOME=/usr/local/pulsar + pulsar_dir=/usr/local/pulsar ++ cat + mtls_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls brokerClientTlsEnabled=true brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_auto_stop/ca.cert.pem brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/changefeed_auto_stop/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/changefeed_auto_stop/broker_client.key-pk8.pem"} brokerServicePortTls=6651 webServicePortTls=8443 tlsTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_auto_stop/ca.cert.pem tlsCertificateFilePath=/tmp/tidb_cdc_test/changefeed_auto_stop/server.cert.pem tlsKeyFilePath=/tmp/tidb_cdc_test/changefeed_auto_stop/server.key-pk8.pem tlsRequireTrustedClientCertOnConnect=true tlsAllowInsecureConnection=false tlsCertRefreshCheckDurationSec=300' ++ cat + normal_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' ++ cat + mtls_client_conf=' webServiceUrl=https://localhost:8443/ brokerServiceUrl=pulsar+ssl://localhost:6651/ authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls authParams=tlsCertFile:/tmp/tidb_cdc_test/changefeed_auto_stop/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/changefeed_auto_stop/broker_client.key-pk8.pem tlsTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_auto_stop/ca.cert.pem' ++ cat + oauth_client_conf=' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/ authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 authParams={"privateKey":"/tmp/tidb_cdc_test/changefeed_auto_stop/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}' ++ cat + oauth_conf=' authenticationEnabled=true authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/changefeed_auto_stop/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"} tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag==' ++ cat + credential_json=' { "client_id":"1234", "client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL", "audience":"cdc-api-uri", "issuer_url":"http://localhost:9096", "type": "client_credentials" }' ++ cat + cert_server_conf='[ req ] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn [ v3_ext ] authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE keyUsage=critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth subjectAltName=@alt_names [ dn ] CN = server [ alt_names ] DNS.1 = localhost IP.1 = 127.0.0.1' + echo ' webServiceUrl=http://localhost:8080/ brokerServiceUrl=pulsar://localhost:6650/' + cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/changefeed_auto_stop/pulsar_standalone.conf + pulsar_port=6650 + '[' normal == mtls ']' + '[' normal == oauth ']' + echo 'no cluster type specified, using default configuration.' no cluster type specified, using default configuration. ++ date + echo '[Mon May 6 19:54:21 CST 2024] <<<<<< START pulsar cluster in normal mode in changefeed_auto_stop case >>>>>>' [Mon May 6 19:54:21 CST 2024] <<<<<< START pulsar cluster in normal mode in changefeed_auto_stop case >>>>>> + echo 'Waiting for pulsar port to be ready...' Waiting for pulsar port to be ready... + i=0 + nc -z localhost 6650 + /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/changefeed_auto_stop/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/changefeed_auto_stop/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/changefeed_auto_stop/pulsar-bookie + i=1 + '[' 1 -gt 20 ']' + sleep 2 0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0{"level":"warn","ts":"2024-05-06T19:54:23.083885+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0010c2000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2024-05-06T19:54:23.083947+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-05-06T19:54:23.085832+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000aad6c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2024-05-06T19:54:23.085895+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-05-06T19:54:23.149678+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0012ec000/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":"2024-05-06T19:54:23.149746+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} + nc -z localhost 6650 + i=2 + '[' 2 -gt 20 ']' + sleep 2 check diff failed 3-th time, retry later + nc -z localhost 6650 + echo 'Waiting for pulsar namespace to be ready...' Waiting for pulsar namespace to be ready... + i=0 + /usr/local/pulsar/bin/pulsar-admin namespaces list public =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/run.sh using Sink-Type: pulsar... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status + source /home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/synced_status + CDC_BINARY=cdc.test + SINK_TYPE=pulsar + CDC_COUNT=3 + DB_COUNT=4 + trap stop_tidb_cluster EXIT + run_normal_case_and_unavailable_pd conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status The 1 times to try to start tidb cluster... check diff failed 4-th time, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:14 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:15 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0{"level":"warn","ts":"2024-05-06T19:54:27.100002+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0010c2000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:54:27.101249+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000aad6c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:54:27.161831+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0012ec000/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} Tenant not found Reason: Tenant not found + i=1 + '[' 1 -gt 20 ']' + sleep 2 check diff successfully wait process cdc.test exit for 1-th time... start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Mon May 6 19:54:29 CST 2024] <<<<<< run test case sink_hang success! >>>>>> + /usr/local/pulsar/bin/pulsar-admin namespaces list public Verifying downstream PD is started... public/default ++ date + echo '[Mon May 6 19:54:32 CST 2024] <<<<<< pulsar is ready >>>>>>' [Mon May 6 19:54:32 CST 2024] <<<<<< pulsar is ready >>>>>> 0 0 0 0 0 0 0 0 --:--:-- 0:00:17 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:18 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:21 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:22 --:--:-- 0{"level":"warn","ts":"2024-05-06T19:54:33.101804+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0010c2000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:54:33.102127+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000aad6c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:54:33.163798+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0012ec000/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} [Mon May 6 19:54:33 CST 2024] <<<<<< START Pulsar consumer in changefeed_auto_stop case >>>>>> check_changefeed_state http://127.0.0.1:2379 be84c5e3-aa2a-4e9f-93d0-2769118c50dd normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=be84c5e3-aa2a-4e9f-93d0-2769118c50dd + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c be84c5e3-aa2a-4e9f-93d0-2769118c50dd -s Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release {"level":"warn","ts":1714996473.2578063,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002f83880/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714996473.2578611,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1714996473.2706785,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002cfca80/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1714996473.27072,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1714996473.272865,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00220ce00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714996473.2729235,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} + info='{ "upstream_id": 7365853613935670194, "namespace": "default", "id": "be84c5e3-aa2a-4e9f-93d0-2769118c50dd", "state": "normal", "checkpoint_tso": 449576029555785729, "checkpoint_time": "2024-05-06 19:54:10.637", "error": null }' + echo '{ "upstream_id": 7365853613935670194, "namespace": "default", "id": "be84c5e3-aa2a-4e9f-93d0-2769118c50dd", "state": "normal", "checkpoint_tso": 449576029555785729, "checkpoint_time": "2024-05-06 19:54:10.637", "error": null }' { "upstream_id": 7365853613935670194, "namespace": "default", "id": "be84c5e3-aa2a-4e9f-93d0-2769118c50dd", "state": "normal", "checkpoint_tso": 449576029555785729, "checkpoint_time": "2024-05-06 19:54:10.637", "error": null } ++ echo '{' '"upstream_id":' 7365853613935670194, '"namespace":' '"default",' '"id":' '"be84c5e3-aa2a-4e9f-93d0-2769118c50dd",' '"state":' '"normal",' '"checkpoint_tso":' 449576029555785729, '"checkpoint_time":' '"2024-05-06' '19:54:10.637",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7365853613935670194, '"namespace":' '"default",' '"id":' '"be84c5e3-aa2a-4e9f-93d0-2769118c50dd",' '"state":' '"normal",' '"checkpoint_tso":' 449576029555785729, '"checkpoint_time":' '"2024-05-06' '19:54:10.637",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully table changefeed_auto_stop_1.usertable not exists for 1-th check, retry later Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table changefeed_auto_stop_1.usertable not exists for 2-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table changefeed_auto_stop_1.usertable exists table changefeed_auto_stop_2.usertable not exists for 1-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:23 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:24 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:25 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:26 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:27 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:28 --:--:-- 0{"level":"warn","ts":"2024-05-06T19:54:39.102971+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000aad6c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:54:39.10347+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0010c2000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:54:39.164882+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0012ec000/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } table changefeed_auto_stop_2.usertable exists table changefeed_auto_stop_3.usertable not exists for 1-th check, retry later [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } 0 0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 27 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 33 + synced_status='{ "error_msg": "[CDC:ErrPDEtcdAPIError]etcd api call error: context deadline exceeded", "error_code": "CDC:ErrPDEtcdAPIError" }' ++ echo '{' '"error_msg":' '"[CDC:ErrPDEtcdAPIError]etcd' api call error: context deadline 'exceeded",' '"error_code":' '"CDC:ErrPDEtcdAPIError"' '}' ++ jq -r .error_code + error_code=CDC:ErrPDEtcdAPIError + cleanup_process cdc.test wait process cdc.test exit for 1-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table changefeed_auto_stop_3.usertable exists table changefeed_auto_stop_4.usertable not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit + stop_tidb_cluster VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370622d00013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb, pid:18679, start at 2024-05-06 19:54:40.220667273 +0800 CST m=+5.254524371 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:56:40.228 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:54:40.230 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:44:40.230 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370622d00013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb, pid:18679, start at 2024-05-06 19:54:40.220667273 +0800 CST m=+5.254524371 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:56:40.228 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:54:40.230 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:44:40.230 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d370623980015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb, pid:18761, start at 2024-05-06 19:54:40.264962985 +0800 CST m=+5.248050456 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:56:40.272 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:54:40.230 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:44:40.230 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table changefeed_auto_stop_4.usertable not exists for 2-th check, retry later table changefeed_auto_stop_4.usertable exists check diff failed 1-th time, retry later + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.20095.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449576038729252865 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576038729252865 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449576038729252865 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Mon May 6 19:54:47 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2013320135.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Mon May 6 19:54:49 CST 2024] <<<<<< run test case changefeed_auto_stop success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:54:50 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f110d374-5f46-4d42-ab83-b652ca2fceee {"id":"f110d374-5f46-4d42-ab83-b652ca2fceee","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996487} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc15c8c3d f110d374-5f46-4d42-ab83-b652ca2fceee /tidb/cdc/default/default/upstream/7365853747672951305 {"id":7365853747672951305,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f110d374-5f46-4d42-ab83-b652ca2fceee {"id":"f110d374-5f46-4d42-ab83-b652ca2fceee","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996487} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc15c8c3d f110d374-5f46-4d42-ab83-b652ca2fceee /tidb/cdc/default/default/upstream/7365853747672951305 {"id":7365853747672951305,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f110d374-5f46-4d42-ab83-b652ca2fceee {"id":"f110d374-5f46-4d42-ab83-b652ca2fceee","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996487} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc15c8c3d f110d374-5f46-4d42-ab83-b652ca2fceee /tidb/cdc/default/default/upstream/7365853747672951305 {"id":7365853747672951305,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449576038729252865 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.20191.out cli changefeed create --start-ts=449576038729252865 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7365853747672951305,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-06T19:54:50.628882533+08:00","start_ts":449576038729252865,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576038729252865,"checkpoint_ts":449576038729252865,"checkpoint_time":"2024-05-06 19:54:45.631"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... + set +x ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 1488 0 --:--:-- --:--:-- --:--:-- 1493 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-06 19:54:52.030","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-05-06 19:54:52.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '19:54:52.030","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-06' '19:54:52.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '19:54:52.030","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-06' '19:54:52.000","info":"Data' syncing is 'finished"}' ++ jq -r .sink_checkpoint_ts + sink_checkpoint_ts='2024-05-06 19:54:52.030' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '19:54:52.030","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-06' '19:54:52.000","info":"Data' syncing is 'finished"}' ++ jq -r .puller_resolved_ts + puller_resolved_ts='1970-01-01 08:00:00.000' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '19:54:52.030","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-06' '19:54:52.000","info":"Data' syncing is 'finished"}' ++ jq -r .last_synced_ts + last_synced_ts='1970-01-01 08:00:00.000' + '[' true '!=' true ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' ++ date '+%Y-%m-%d %H:%M:%S' + current='2024-05-06 19:54:52' + echo 'sink_checkpoint_ts is 2024-05-06' 19:54:52.030 sink_checkpoint_ts is 2024-05-06 19:54:52.030 ++ date -d '2024-05-06 19:54:52.030' +%s + checkpoint_timestamp=1714996492 ++ date -d '2024-05-06 19:54:52' +%s + current_timestamp=1714996492 + '[' 0 -gt 300 ']' + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + run_case_with_unavailable_tikv conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status_with_redo + mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 3940 0 --:--:-- --:--:-- --:--:-- 3983 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-06 19:54:59.030","puller_resolved_ts":"2024-05-06 19:54:52.380","last_synced_ts":"2024-05-06 19:54:52.880","now_ts":"2024-05-06 19:54:59.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '19:54:59.030","puller_resolved_ts":"2024-05-06' '19:54:52.380","last_synced_ts":"2024-05-06' '19:54:52.880","now_ts":"2024-05-06' '19:54:59.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '19:54:59.030","puller_resolved_ts":"2024-05-06' '19:54:52.380","last_synced_ts":"2024-05-06' '19:54:52.880","now_ts":"2024-05-06' '19:54:59.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3707d2ec0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6, pid:14560, start at 2024-05-06 19:55:07.868887097 +0800 CST m=+5.330616907 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:57:07.874 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:55:07.835 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:45:07.835 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3707d2ec0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6, pid:14560, start at 2024-05-06 19:55:07.868887097 +0800 CST m=+5.330616907 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:57:07.874 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:55:07.835 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:45:07.835 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3707d524000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6, pid:14631, start at 2024-05-06 19:55:07.9878858 +0800 CST m=+5.395350394 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-19:57:07.994 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:55:07.977 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:45:07.977 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status_with_redo ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.15897.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449576045952368641 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576045952368641 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449576045952368641 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test [Mon May 6 19:55:14 CST 2024] <<<<<< START cdc server in synced_status_with_redo case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.1593915941.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:55:17 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9b57dc95-aa61-4408-860e-e94d2a2a81df {"id":"9b57dc95-aa61-4408-860e-e94d2a2a81df","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996515} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1cc153a 9b57dc95-aa61-4408-860e-e94d2a2a81df /tidb/cdc/default/default/upstream/7365853879480308443 {"id":7365853879480308443,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9b57dc95-aa61-4408-860e-e94d2a2a81df {"id":"9b57dc95-aa61-4408-860e-e94d2a2a81df","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996515} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1cc153a 9b57dc95-aa61-4408-860e-e94d2a2a81df /tidb/cdc/default/default/upstream/7365853879480308443 {"id":7365853879480308443,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9b57dc95-aa61-4408-860e-e94d2a2a81df {"id":"9b57dc95-aa61-4408-860e-e94d2a2a81df","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996515} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc1cc153a 9b57dc95-aa61-4408-860e-e94d2a2a81df /tidb/cdc/default/default/upstream/7365853879480308443 {"id":7365853879480308443,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449576045952368641 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.15999.out cli changefeed create --start-ts=449576045952368641 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7365853879480308443,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-06T19:55:18.184719973+08:00","start_ts":449576045952368641,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576045952368641,"checkpoint_ts":449576045952368641,"checkpoint_time":"2024-05-06 19:55:13.185"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 + kill_tikv ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo + info='jenkins 13878 19.5 0.4 4067452 1892304 ? Sl 19:54 0:05 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20160 --status-addr 127.0.0.1:20181 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tikv1.log --log-level debug -C /tmp/tidb_cdc_test/synced_status_with_redo/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status_with_redo/tikv1 jenkins 13879 18.9 0.4 4057720 1868248 ? Sl 19:54 0:05 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20161 --status-addr 127.0.0.1:20182 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tikv2.log --log-level debug -C /tmp/tidb_cdc_test/synced_status_with_redo/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status_with_redo/tikv2 jenkins 13880 26.5 0.4 4097148 1932404 ? Sl 19:54 0:07 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20162 --status-addr 127.0.0.1:20183 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tikv3.log --log-level debug -C /tmp/tidb_cdc_test/synced_status_with_redo/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status_with_redo/tikv3 jenkins 13882 26.2 0.4 4096636 1923624 ? Sl 19:54 0:07 tikv-server --pd 127.0.0.1:2479 -A 127.0.0.1:21160 --status-addr 127.0.0.1:21180 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tikv_down.log --log-level debug -C /tmp/tidb_cdc_test/synced_status_with_redo/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status_with_redo/tikv_down' ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 1233 0 --:--:-- --:--:-- --:--:-- 1239 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-06 19:55:25.585","puller_resolved_ts":"2024-05-06 19:55:19.634","last_synced_ts":"2024-05-06 19:55:20.135","now_ts":"2024-05-06 19:55:27.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '19:55:25.585","puller_resolved_ts":"2024-05-06' '19:55:19.634","last_synced_ts":"2024-05-06' '19:55:20.135","now_ts":"2024-05-06' '19:55:27.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '19:55:25.585","puller_resolved_ts":"2024-05-06' '19:55:19.634","last_synced_ts":"2024-05-06' '19:55:20.135","now_ts":"2024-05-06' '19:55:27.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2669 0 --:--:-- --:--:-- --:--:-- 2695 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-06 19:57:09.130","puller_resolved_ts":"2024-05-06 19:57:01.130","last_synced_ts":"2024-05-06 19:54:52.880","now_ts":"2024-05-06 19:57:10.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '19:57:09.130","puller_resolved_ts":"2024-05-06' '19:57:01.130","last_synced_ts":"2024-05-06' '19:54:52.880","now_ts":"2024-05-06' '19:57:10.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' + kill_pd ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 17921 8.5 0.0 13643284 143672 ? Sl 19:54 0:13 pd-server --advertise-client-urls http://127.0.0.1:2379 --client-urls http://0.0.0.0:2379 --advertise-peer-urls http://127.0.0.1:2380 --peer-urls http://0.0.0.0:2380 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/pd1.log --data-dir /tmp/tidb_cdc_test/synced_status/pd1 --name=pd1 --initial-cluster=pd1=http://127.0.0.1:2380 jenkins 17985 5.5 0.0 13449108 138536 ? Sl 19:54 0:08 pd-server --advertise-client-urls http://127.0.0.1:2479 --client-urls http://0.0.0.0:2479 --advertise-peer-urls http://127.0.0.1:2480 --peer-urls http://0.0.0.0:2480 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/down_pd.log --data-dir /tmp/tidb_cdc_test/synced_status/down_pd' ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 + sleep 20 {"level":"warn","ts":1714996635.258961,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002d9afc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714996635.259041,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1714996635.2994008,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00157d180/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1714996635.2994742,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1714996636.1564245,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0021f7340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714996636.1564965,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-05-06T19:57:20.029744+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000a57500/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:57:20.029977+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000eb9180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:57:20.084334+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0010de1c0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:57:26.030613+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000a57500/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:57:26.031585+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000eb9180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:57:26.085907+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0010de1c0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0{"level":"warn","ts":"2024-05-06T19:57:32.032307+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000eb9180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:57:32.032618+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000a57500/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:57:32.087175+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0010de1c0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0{"level":"warn","ts":"2024-05-06T19:57:38.033621+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000eb9180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:57:38.033771+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000a57500/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:57:38.088249+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0010de1c0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0{"level":"warn","ts":"2024-05-06T19:57:40.017214+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000a57500/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2024-05-06T19:57:40.017316+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-05-06T19:57:40.018724+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000eb9180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2024-05-06T19:57:40.018764+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-05-06T19:57:40.074371+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0010de1c0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":"2024-05-06T19:57:40.07443+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0{"level":"warn","ts":"2024-05-06T19:57:44.035231+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000eb9180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:57:44.035439+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000a57500/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:57:44.089223+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0010de1c0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 8882 0 --:--:-- --:--:-- --:--:-- 8817 100 723 100 723 0 0 8874 0 --:--:-- --:--:-- --:--:-- 8817 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-06 19:55:26.584","puller_resolved_ts":"2024-05-06 19:55:26.584","last_synced_ts":"2024-05-06 19:55:20.135","now_ts":"2024-05-06 19:57:37.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '19:55:26.584","puller_resolved_ts":"2024-05-06' '19:55:26.584","last_synced_ts":"2024-05-06' '19:55:20.135","now_ts":"2024-05-06' '19:57:37.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '19:55:26.584","puller_resolved_ts":"2024-05-06' '19:55:26.584","last_synced_ts":"2024-05-06' '19:55:20.135","now_ts":"2024-05-06' '19:57:37.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tidb conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status_with_redo + mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory 0 0 0 0 0 0 0 0 --:--:-- 0:00:14 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:15 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:17 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:18 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0{"level":"warn","ts":"2024-05-06T19:57:50.03623+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000eb9180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:57:50.036986+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000a57500/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:57:50.090001+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0010de1c0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0{"level":"warn","ts":1714996670.2594407,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002d9afc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714996670.2594965,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1714996670.3007324,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00157d180/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1714996670.3007739,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1714996671.1585715,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0021f7340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714996671.1586251,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... 0 0 0 0 0 0 0 0 --:--:-- 0:00:21 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:22 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:23 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:24 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:25 --:--:-- 0{"level":"warn","ts":"2024-05-06T19:57:56.037409+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000eb9180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:57:56.038585+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000a57500/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:57:56.090772+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0010de1c0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) 0 0 0 0 0 0 0 0 --:--:-- 0:00:26 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:27 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:28 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 27 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 33 + synced_status='{ "error_msg": "[CDC:ErrPDEtcdAPIError]etcd api call error: context deadline exceeded", "error_code": "CDC:ErrPDEtcdAPIError" }' ++ jq -r .error_code ++ echo '{' '"error_msg":' '"[CDC:ErrPDEtcdAPIError]etcd' api call error: context deadline 'exceeded",' '"error_code":' '"CDC:ErrPDEtcdAPIError"' '}' + error_code=CDC:ErrPDEtcdAPIError + cleanup_process cdc.test ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... {"level":"warn","ts":"2024-05-06T19:58:02.038915+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000eb9180/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:58:02.040342+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000a57500/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-05-06T19:58:02.091761+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0010de1c0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit + stop_tidb_cluster ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3712868c0005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6, pid:17220, start at 2024-05-06 19:58:03.175315704 +0800 CST m=+5.162462450 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:00:03.181 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:58:03.171 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:48:03.171 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3712868c0005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6, pid:17220, start at 2024-05-06 19:58:03.175315704 +0800 CST m=+5.162462450 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:00:03.181 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:58:03.171 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:48:03.171 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d371285b0000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6, pid:17295, start at 2024-05-06 19:58:03.12522544 +0800 CST m=+5.061541164 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:00:03.131 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:58:03.116 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:48:03.116 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status_with_redo ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.18575.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449576091417575425 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576091417575425 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449576091417575425 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test [Mon May 6 19:58:08 CST 2024] <<<<<< START cdc server in synced_status_with_redo case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.1861618618.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + run_case_with_unavailable_tikv conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:58:11 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2f09a2ae-a78c-416d-be80-38ae888b93bc {"id":"2f09a2ae-a78c-416d-be80-38ae888b93bc","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996688} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc477a037 2f09a2ae-a78c-416d-be80-38ae888b93bc /tidb/cdc/default/default/upstream/7365854628455396205 {"id":7365854628455396205,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2f09a2ae-a78c-416d-be80-38ae888b93bc {"id":"2f09a2ae-a78c-416d-be80-38ae888b93bc","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996688} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc477a037 2f09a2ae-a78c-416d-be80-38ae888b93bc /tidb/cdc/default/default/upstream/7365854628455396205 {"id":7365854628455396205,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2f09a2ae-a78c-416d-be80-38ae888b93bc {"id":"2f09a2ae-a78c-416d-be80-38ae888b93bc","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996688} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc477a037 2f09a2ae-a78c-416d-be80-38ae888b93bc /tidb/cdc/default/default/upstream/7365854628455396205 {"id":7365854628455396205,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449576091417575425 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.18673.out cli changefeed create --start-ts=449576091417575425 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7365854628455396205,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-06T19:58:11.630655747+08:00","start_ts":449576091417575425,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576091417575425,"checkpoint_ts":449576091417575425,"checkpoint_time":"2024-05-06 19:58:06.621"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table test.t1 exists + sleep 5 Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + kill_tidb ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo + info='jenkins 17216 3.3 0.0 2357740 195788 ? Sl 19:57 0:00 tidb-server -P 4000 -config /tmp/tidb_cdc_test/synced_status_with_redo/tidb-config-1714996678003909961.toml --store tikv --path 127.0.0.1:2379 --status=10080 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tidb.log jenkins 17220 12.3 0.0 2440888 237296 ? Sl 19:57 0:02 tidb-server -P 4001 -config /tmp/tidb_cdc_test/synced_status_with_redo/tidb-config-1714996678007097060.toml --store tikv --path 127.0.0.1:2379 --status=10081 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tidb_other.log jenkins 17295 11.6 0.0 2662916 255924 ? Sl 19:57 0:02 tidb-server -P 3306 -config /tmp/tidb_cdc_test/synced_status_with_redo/tidb-config-1714996678056837545.toml --store tikv --path 127.0.0.1:2479 --status=20080 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tidb_down.log' ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 3788 0 --:--:-- --:--:-- --:--:-- 3796 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-06 19:58:18.371","puller_resolved_ts":"2024-05-06 19:58:13.072","last_synced_ts":"2024-05-06 19:58:13.571","now_ts":"2024-05-06 19:58:21.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '19:58:18.371","puller_resolved_ts":"2024-05-06' '19:58:13.072","last_synced_ts":"2024-05-06' '19:58:13.571","now_ts":"2024-05-06' '19:58:21.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '19:58:18.371","puller_resolved_ts":"2024-05-06' '19:58:13.072","last_synced_ts":"2024-05-06' '19:58:13.571","now_ts":"2024-05-06' '19:58:21.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3713c31c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb, pid:21374, start at 2024-05-06 19:58:23.453449962 +0800 CST m=+5.384779179 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:00:23.460 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:58:23.431 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:48:23.431 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3713c31c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb, pid:21374, start at 2024-05-06 19:58:23.453449962 +0800 CST m=+5.384779179 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:00:23.460 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:58:23.431 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:48:23.431 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3713c2880008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb, pid:21450, start at 2024-05-06 19:58:23.406564081 +0800 CST m=+5.282229134 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:00:23.413 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-19:58:23.394 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:48:23.394 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.22781.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449576097213317121 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576097213317121 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449576097213317121 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Mon May 6 19:58:30 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2281522817.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 11:58:33 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0cbbb83a-7ce7-4176-9f19-2e8b70f7ef60 {"id":"0cbbb83a-7ce7-4176-9f19-2e8b70f7ef60","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996710} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc4c7ef39 0cbbb83a-7ce7-4176-9f19-2e8b70f7ef60 /tidb/cdc/default/default/upstream/7365854715789174332 {"id":7365854715789174332,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0cbbb83a-7ce7-4176-9f19-2e8b70f7ef60 {"id":"0cbbb83a-7ce7-4176-9f19-2e8b70f7ef60","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996710} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc4c7ef39 0cbbb83a-7ce7-4176-9f19-2e8b70f7ef60 /tidb/cdc/default/default/upstream/7365854715789174332 {"id":7365854715789174332,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0cbbb83a-7ce7-4176-9f19-2e8b70f7ef60 {"id":"0cbbb83a-7ce7-4176-9f19-2e8b70f7ef60","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996710} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc4c7ef39 0cbbb83a-7ce7-4176-9f19-2e8b70f7ef60 /tidb/cdc/default/default/upstream/7365854715789174332 {"id":7365854715789174332,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449576097213317121 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.22874.out cli changefeed create --start-ts=449576097213317121 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7365854715789174332,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-06T19:58:33.74012766+08:00","start_ts":449576097213317121,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576097213317121,"checkpoint_ts":449576097213317121,"checkpoint_time":"2024-05-06 19:58:28.730"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 + kill_tikv ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 20750 19.7 0.4 4060280 1866780 ? Sl 19:58 0:05 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20160 --status-addr 127.0.0.1:20181 --log-file /tmp/tidb_cdc_test/synced_status/tikv1.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv1 jenkins 20751 28.1 0.4 4101240 1932036 ? Sl 19:58 0:07 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20161 --status-addr 127.0.0.1:20182 --log-file /tmp/tidb_cdc_test/synced_status/tikv2.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv2 jenkins 20752 20.3 0.4 4064888 1863436 ? Sl 19:58 0:05 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20162 --status-addr 127.0.0.1:20183 --log-file /tmp/tidb_cdc_test/synced_status/tikv3.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv3 jenkins 20754 26.5 0.4 4085876 1922308 ? Sl 19:58 0:06 tikv-server --pd 127.0.0.1:2479 -A 127.0.0.1:21160 --status-addr 127.0.0.1:21180 --log-file /tmp/tidb_cdc_test/synced_status/tikv_down.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv_down' ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 3962 0 --:--:-- --:--:-- --:--:-- 3983 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-06 19:58:41.131","puller_resolved_ts":"2024-05-06 19:58:35.231","last_synced_ts":"2024-05-06 19:58:35.280","now_ts":"2024-05-06 19:58:42.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '19:58:41.131","puller_resolved_ts":"2024-05-06' '19:58:35.231","last_synced_ts":"2024-05-06' '19:58:35.280","now_ts":"2024-05-06' '19:58:42.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '19:58:41.131","puller_resolved_ts":"2024-05-06' '19:58:35.231","last_synced_ts":"2024-05-06' '19:58:35.280","now_ts":"2024-05-06' '19:58:42.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2674 0 --:--:-- --:--:-- --:--:-- 2695 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-06 20:00:29.521","puller_resolved_ts":"2024-05-06 20:00:22.521","last_synced_ts":"2024-05-06 19:58:13.571","now_ts":"2024-05-06 20:00:31.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '20:00:29.521","puller_resolved_ts":"2024-05-06' '20:00:22.521","last_synced_ts":"2024-05-06' '19:58:13.571","now_ts":"2024-05-06' '20:00:31.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '20:00:29.521","puller_resolved_ts":"2024-05-06' '20:00:22.521","last_synced_ts":"2024-05-06' '19:58:13.571","now_ts":"2024-05-06' '20:00:31.000","info":"Data' syncing is 'finished"}' ++ jq -r .info + info='Data syncing is finished' + target_message='Data syncing is finished' + '[' 'Data syncing is finished' '!=' 'Data syncing is finished' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_failpoint conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status_with_redo + mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d371d0c6c001a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6, pid:19781, start at 2024-05-06 20:00:55.609333062 +0800 CST m=+5.156068098 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:02:55.617 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-20:00:55.579 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:50:55.579 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d371d0c6c001a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6, pid:19781, start at 2024-05-06 20:00:55.609333062 +0800 CST m=+5.156068098 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:02:55.617 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-20:00:55.579 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:50:55.579 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d371d0de80015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-jptl6, pid:19871, start at 2024-05-06 20:00:55.721426991 +0800 CST m=+5.205417701 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:02:55.728 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-20:00:55.724 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:50:55.724 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status_with_redo + export 'GO_FAILPOINTS=github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.21270.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449576136626667521 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576136626667521 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449576136626667521 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test [Mon May 6 20:01:00 CST 2024] <<<<<< START cdc server in synced_status_with_redo case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.2130721309.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 12:01:03 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/32432c78-1f89-4bf1-8347-25916b8cb292 {"id":"32432c78-1f89-4bf1-8347-25916b8cb292","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996860} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc71d0134 32432c78-1f89-4bf1-8347-25916b8cb292 /tidb/cdc/default/default/upstream/7365855368611095133 {"id":7365855368611095133,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/32432c78-1f89-4bf1-8347-25916b8cb292 {"id":"32432c78-1f89-4bf1-8347-25916b8cb292","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996860} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc71d0134 32432c78-1f89-4bf1-8347-25916b8cb292 /tidb/cdc/default/default/upstream/7365855368611095133 {"id":7365855368611095133,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/32432c78-1f89-4bf1-8347-25916b8cb292 {"id":"32432c78-1f89-4bf1-8347-25916b8cb292","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996860} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc71d0134 32432c78-1f89-4bf1-8347-25916b8cb292 /tidb/cdc/default/default/upstream/7365855368611095133 {"id":7365855368611095133,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449576136626667521 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.21369.out cli changefeed create --start-ts=449576136626667521 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7365855368611095133,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-06T20:01:04.079337987+08:00","start_ts":449576136626667521,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576136626667521,"checkpoint_ts":449576136626667521,"checkpoint_time":"2024-05-06 20:00:59.080"} PASS coverage: 2.7% of statements in github.com/pingcap/tiflow/... ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 8874 0 --:--:-- --:--:-- --:--:-- 8925 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-06 19:58:42.131","puller_resolved_ts":"2024-05-06 19:58:42.131","last_synced_ts":"2024-05-06 19:58:35.280","now_ts":"2024-05-06 20:00:52.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ jq .synced ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '19:58:42.131","puller_resolved_ts":"2024-05-06' '19:58:42.131","last_synced_ts":"2024-05-06' '19:58:35.280","now_ts":"2024-05-06' '20:00:52.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' + status=false + '[' false '!=' false ']' ++ jq -r .info ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '19:58:42.131","puller_resolved_ts":"2024-05-06' '19:58:42.131","last_synced_ts":"2024-05-06' '19:58:35.280","now_ts":"2024-05-06' '20:00:52.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tidb conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... + set +x + sleep 20 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d371e3fcc0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb, pid:24065, start at 2024-05-06 20:01:15.280643608 +0800 CST m=+5.134870796 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:03:15.287 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-20:01:15.251 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:51:15.251 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d371e3fcc0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb, pid:24065, start at 2024-05-06 20:01:15.280643608 +0800 CST m=+5.134870796 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:03:15.287 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-20:01:15.251 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:51:15.251 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d371e408c0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb, pid:24155, start at 2024-05-06 20:01:15.321513339 +0800 CST m=+5.126877769 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:03:15.328 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-20:01:15.299 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:51:15.299 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.25484.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449576141783302145 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576141783302145 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449576141783302145 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Mon May 6 20:01:20 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2552425526.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 12:01:23 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f8f04579-6967-42c8-a7cd-4100cee813d0 {"id":"f8f04579-6967-42c8-a7cd-4100cee813d0","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996880} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc7681734 f8f04579-6967-42c8-a7cd-4100cee813d0 /tidb/cdc/default/default/upstream/7365855460757060160 {"id":7365855460757060160,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f8f04579-6967-42c8-a7cd-4100cee813d0 {"id":"f8f04579-6967-42c8-a7cd-4100cee813d0","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996880} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc7681734 f8f04579-6967-42c8-a7cd-4100cee813d0 /tidb/cdc/default/default/upstream/7365855460757060160 {"id":7365855460757060160,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f8f04579-6967-42c8-a7cd-4100cee813d0 {"id":"f8f04579-6967-42c8-a7cd-4100cee813d0","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714996880} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dc7681734 f8f04579-6967-42c8-a7cd-4100cee813d0 /tidb/cdc/default/default/upstream/7365855460757060160 {"id":7365855460757060160,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449576141783302145 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.25577.out cli changefeed create --start-ts=449576141783302145 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7365855460757060160,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-06T20:01:23.790814185+08:00","start_ts":449576141783302145,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576141783302145,"checkpoint_ts":449576141783302145,"checkpoint_time":"2024-05-06 20:01:18.751"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 4150 0 --:--:-- --:--:-- --:--:-- 4155 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-06 20:00:59.080","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-05-06 20:01:25.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '20:00:59.080","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-06' '20:01:25.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '20:00:59.080","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-06' '20:01:25.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + export GO_FAILPOINTS= + GO_FAILPOINTS= + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit + stop_tidb_cluster table test.t1 exists + sleep 5 + kill_tidb ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 24061 3.7 0.0 3030224 190768 ? Sl 20:01 0:00 tidb-server -P 4000 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1714996870136145754.toml --store tikv --path 127.0.0.1:2379 --status=10080 --log-file /tmp/tidb_cdc_test/synced_status/tidb.log jenkins 24065 13.0 0.0 2950412 253904 ? Sl 20:01 0:02 tidb-server -P 4001 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1714996870138914453.toml --store tikv --path 127.0.0.1:2379 --status=10081 --log-file /tmp/tidb_cdc_test/synced_status/tidb_other.log jenkins 24155 14.0 0.0 3254048 250904 ? Sl 20:01 0:03 tidb-server -P 3306 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1714996870186932386.toml --store tikv --path 127.0.0.1:2479 --status=20080 --log-file /tmp/tidb_cdc_test/synced_status/tidb_down.log' ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 2914 0 --:--:-- --:--:-- --:--:-- 2927 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-06 20:01:32.202","puller_resolved_ts":"2024-05-06 20:01:25.301","last_synced_ts":"2024-05-06 20:01:25.751","now_ts":"2024-05-06 20:01:32.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '20:01:32.202","puller_resolved_ts":"2024-05-06' '20:01:25.301","last_synced_ts":"2024-05-06' '20:01:25.751","now_ts":"2024-05-06' '20:01:32.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '20:01:32.202","puller_resolved_ts":"2024-05-06' '20:01:25.301","last_synced_ts":"2024-05-06' '20:01:25.751","now_ts":"2024-05-06' '20:01:32.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 + check_logs /tmp/tidb_cdc_test/synced_status_with_redo ++ date + echo '[Mon May 6 20:01:38 CST 2024] <<<<<< run test case synced_status_with_redo success! >>>>>>' [Mon May 6 20:01:38 CST 2024] <<<<<< run test case synced_status_with_redo success! >>>>>> + stop_tidb_cluster \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2581 0 --:--:-- --:--:-- --:--:-- 2600 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-06 20:03:42.301","puller_resolved_ts":"2024-05-06 20:03:34.301","last_synced_ts":"2024-05-06 20:01:25.751","now_ts":"2024-05-06 20:03:43.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '20:03:42.301","puller_resolved_ts":"2024-05-06' '20:03:34.301","last_synced_ts":"2024-05-06' '20:01:25.751","now_ts":"2024-05-06' '20:03:43.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-06' '20:03:42.301","puller_resolved_ts":"2024-05-06' '20:03:34.301","last_synced_ts":"2024-05-06' '20:01:25.751","now_ts":"2024-05-06' '20:03:43.000","info":"Data' syncing is 'finished"}' ++ jq -r .info + info='Data syncing is finished' + target_message='Data syncing is finished' + '[' 'Data syncing is finished' '!=' 'Data syncing is finished' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_failpoint conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Starting Downstream PD... Release Version: v8.1.0-alpha-20-g13eaf8aaa Edition: Community Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d Git Branch: release-8.1 UTC Build Time: 2024-04-30 07:22:55 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 8.1.0 Edition: Community Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8 Git Commit Branch: release-8.1 UTC Build Time: 2024-04-30 06:15:40 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v8.1.0-alpha-106-gc8e6662d10 Edition: Community Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991 Git Branch: release-8.1 UTC Build Time: 2024-05-06 10:39:26 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3728b0f40015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb, pid:26680, start at 2024-05-06 20:04:06.35778332 +0800 CST m=+5.156057885 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:06:06.365 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-20:04:06.333 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:54:06.333 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3728b0f40015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb, pid:26680, start at 2024-05-06 20:04:06.35778332 +0800 CST m=+5.156057885 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:06:06.365 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-20:04:06.333 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:54:06.333 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 196 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63d3728b184000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-72-86stb, pid:26771, start at 2024-05-06 20:04:06.381896915 +0800 CST m=+5.124705858 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240506-20:06:06.388 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240506-20:04:06.369 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240506-19:54:06.369 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v8.1.0-alpha-20-geb96e7201 Edition: Community Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c Git Branch: HEAD UTC Build Time: 2024-05-05 06:18:35 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Compiler: clang++ 13.0.0 Raft Proxy Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79 Git Commit Branch: HEAD UTC Build Time: 2024-05-05 06:22:21 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status + export 'GO_FAILPOINTS=github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.28158.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449576186631421953 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...' + echo 449576186631421953 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449576186631421953 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Mon May 6 20:04:11 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2820628208.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 06 May 2024 12:04:14 GMT < Content-Length: 828 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9d1746c5-40e0-456d-a490-237f363fd5b9 {"id":"9d1746c5-40e0-456d-a490-237f363fd5b9","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714997051} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dca023a38 9d1746c5-40e0-456d-a490-237f363fd5b9 /tidb/cdc/default/default/upstream/7365856183764708544 {"id":7365856183764708544,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9d1746c5-40e0-456d-a490-237f363fd5b9 {"id":"9d1746c5-40e0-456d-a490-237f363fd5b9","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714997051} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dca023a38 9d1746c5-40e0-456d-a490-237f363fd5b9 /tidb/cdc/default/default/upstream/7365856183764708544 {"id":7365856183764708544,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9d1746c5-40e0-456d-a490-237f363fd5b9 {"id":"9d1746c5-40e0-456d-a490-237f363fd5b9","address":"127.0.0.1:8300","version":"v8.1.0-alpha-23-ge1a27adc6","git-hash":"e1a27adc69acc9d50db0c3fea2945bbd6d983e30","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1714997051} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f4dca023a38 9d1746c5-40e0-456d-a490-237f363fd5b9 /tidb/cdc/default/default/upstream/7365856183764708544 {"id":7365856183764708544,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449576186631421953 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.28264.out cli changefeed create --start-ts=449576186631421953 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7365856183764708544,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-06T20:04:14.854898106+08:00","start_ts":449576186631421953,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v8.1.0-alpha-23-ge1a27adc6","resolved_ts":449576186631421953,"checkpoint_ts":449576186631421953,"checkpoint_time":"2024-05-06 20:04:09.833"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... + set +x + sleep 20 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 6637 0 --:--:-- --:--:-- --:--:-- 6694 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-06 20:04:09.833","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-05-06 20:04:36.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '20:04:09.833","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-06' '20:04:36.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ jq -r .info ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-06' '20:04:09.833","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-06' '20:04:36.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + export GO_FAILPOINTS= + GO_FAILPOINTS= + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + check_logs /tmp/tidb_cdc_test/synced_status ++ date + echo '[Mon May 6 20:04:47 CST 2024] <<<<<< run test case synced_status success! >>>>>>' [Mon May 6 20:04:47 CST 2024] <<<<<< run test case synced_status success! >>>>>> + stop_tidb_cluster \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-72/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // parallel [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] End of Pipeline Finished: SUCCESS