Git Branch: release-8.1
UTC Build Time: 2024-04-30 07:22:55
Starting Downstream PD...
Release Version: v8.1.0-alpha-20-g13eaf8aaa
Edition: Community
Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d
Git Branch: release-8.1
UTC Build Time: 2024-04-30 07:22:55
Verifying upstream PD is started...
Verifying downstream PD is started...
Starting Upstream TiKV...
TiKV
Release Version: 8.1.0
Edition: Community
Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8
Git Commit Branch: release-8.1
UTC Build Time: 2024-04-30 06:15:40
Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27)
Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored
Profile: dist_release
Starting Downstream TiKV...
TiKV
Release Version: 8.1.0
Edition: Community
Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8
Git Commit Branch: release-8.1
UTC Build Time: 2024-04-30 06:15:40
Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27)
Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored
Profile: dist_release
Starting Upstream TiDB...
Release Version: v8.1.0-alpha-106-gc8e6662d10
Edition: Community
Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991
Git Branch: release-8.1
UTC Build Time: 2024-05-06 10:39:26
GoVersion: go1.21.6
Race Enabled: false
Check Table Before Drop: false
Store: unistore
Starting Downstream TiDB...
Release Version: v8.1.0-alpha-106-gc8e6662d10
Edition: Community
Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991
Git Branch: release-8.1
UTC Build Time: 2024-05-06 10:39:26
GoVersion: go1.21.6
Race Enabled: false
Check Table Before Drop: false
Store: unistore
Verifying Upstream TiDB is started...
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
VARIABLE_NAME VARIABLE_VALUE COMMENT
bootstrapped True Bootstrap flag. Do not delete.
tidb_server_version 196 Bootstrap version. Do not delete.
system_tz Asia/Shanghai TiDB Global System Timezone.
new_collation_enabled True If the new collations are enabled. Do not edit it.
ddl_table_version 3 DDL Table Version. Do not delete.
tikv_gc_leader_uuid 63d47cd8f9c0013 Current GC worker leader UUID. (DO NOT EDIT)
tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-77-3fjdc, pid:11887, start at 2024-05-07 15:27:36.459846625 +0800 CST m=+5.553933636 Host name and pid of current GC leader. (DO NOT EDIT)
tikv_gc_leader_lease 20240507-15:29:36.477 +0800 Current GC worker leader lease. (DO NOT EDIT)
tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used
tikv_gc_enable true Current GC enable status
tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format.
tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format.
tikv_gc_last_run_time 20240507-15:27:36.473 +0800 The time when last GC starts. (DO NOT EDIT)
tikv_gc_safe_point 20240507-15:17:36.473 +0800 All versions after safe point can be accessed. (DO NOT EDIT)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
VARIABLE_NAME VARIABLE_VALUE COMMENT
bootstrapped True Bootstrap flag. Do not delete.
tidb_server_version 196 Bootstrap version. Do not delete.
system_tz Asia/Shanghai TiDB Global System Timezone.
new_collation_enabled True If the new collations are enabled. Do not edit it.
ddl_table_version 3 DDL Table Version. Do not delete.
tikv_gc_leader_uuid 63d47cd8f9c0013 Current GC worker leader UUID. (DO NOT EDIT)
tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-77-3fjdc, pid:11887, start at 2024-05-07 15:27:36.459846625 +0800 CST m=+5.553933636 Host name and pid of current GC leader. (DO NOT EDIT)
tikv_gc_leader_lease 20240507-15:29:36.477 +0800 Current GC worker leader lease. (DO NOT EDIT)
tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used
tikv_gc_enable true Current GC enable status
tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format.
tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format.
tikv_gc_last_run_time 20240507-15:27:36.473 +0800 The time when last GC starts. (DO NOT EDIT)
tikv_gc_safe_point 20240507-15:17:36.473 +0800 All versions after safe point can be accessed. (DO NOT EDIT)
Verifying Downstream TiDB is started...
VARIABLE_NAME VARIABLE_VALUE COMMENT
bootstrapped True Bootstrap flag. Do not delete.
tidb_server_version 196 Bootstrap version. Do not delete.
system_tz Asia/Shanghai TiDB Global System Timezone.
new_collation_enabled True If the new collations are enabled. Do not edit it.
ddl_table_version 3 DDL Table Version. Do not delete.
tikv_gc_leader_uuid 63d47cd9114000f Current GC worker leader UUID. (DO NOT EDIT)
tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-77-3fjdc, pid:11975, start at 2024-05-07 15:27:36.532148129 +0800 CST m=+5.569315552 Host name and pid of current GC leader. (DO NOT EDIT)
tikv_gc_leader_lease 20240507-15:29:36.540 +0800 Current GC worker leader lease. (DO NOT EDIT)
tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used
tikv_gc_enable true Current GC enable status
tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format.
tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format.
tikv_gc_last_run_time 20240507-15:27:36.517 +0800 The time when last GC starts. (DO NOT EDIT)
tikv_gc_safe_point 20240507-15:17:36.517 +0800 All versions after safe point can be accessed. (DO NOT EDIT)
Starting Upstream TiFlash...
TiFlash
Release Version: v8.1.0-alpha-20-geb96e7201
Edition: Community
Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c
Git Branch: HEAD
UTC Build Time: 2024-05-05 06:18:35
Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto
Profile: RELWITHDEBINFO
Compiler: clang++ 13.0.0
Raft Proxy
Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79
Git Commit Branch: HEAD
UTC Build Time: 2024-05-05 06:22:21
Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14)
Storage Engine: tiflash
Prometheus Prefix: tiflash_proxy_
Profile: release
Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored
Verifying Upstream TiFlash is started...
Logging trace to /tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/server.log
Logging errors to /tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/error.log
arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
***************** properties *****************
"threadcount"="4"
"dotransactions"="false"
"mysql.db"="changefeed_auto_stop_1"
"mysql.host"="127.0.0.1"
"workload"="core"
"requestdistribution"="uniform"
"mysql.port"="4000"
"scanproportion"="0"
"recordcount"="20"
"updateproportion"="0"
"operationcount"="0"
"insertproportion"="0"
"mysql.user"="root"
"readproportion"="0"
"readallfields"="true"
**********************************************
Run finished, takes 12.499989ms
INSERT - Takes(s): 0.0, Count: 20, OPS: 2654.1, Avg(us): 2310, Min(us): 1255, Max(us): 5286, 95th(us): 6000, 99th(us): 6000
***************** properties *****************
"requestdistribution"="uniform"
"threadcount"="4"
"insertproportion"="0"
"mysql.db"="changefeed_auto_stop_2"
"mysql.host"="127.0.0.1"
"mysql.user"="root"
"dotransactions"="false"
"readallfields"="true"
"readproportion"="0"
"updateproportion"="0"
"scanproportion"="0"
"operationcount"="0"
"recordcount"="20"
"workload"="core"
"mysql.port"="4000"
**********************************************
Run finished, takes 11.451653ms
INSERT - Takes(s): 0.0, Count: 20, OPS: 3257.9, Avg(us): 2102, Min(us): 1040, Max(us): 4921, 95th(us): 5000, 99th(us): 5000
***************** properties *****************
"dotransactions"="false"
"scanproportion"="0"
"updateproportion"="0"
"requestdistribution"="uniform"
"readproportion"="0"
"mysql.host"="127.0.0.1"
"operationcount"="0"
"mysql.user"="root"
"recordcount"="20"
"workload"="core"
"insertproportion"="0"
"readallfields"="true"
"mysql.port"="4000"
"threadcount"="4"
"mysql.db"="changefeed_auto_stop_3"
**********************************************
Run finished, takes 14.33857ms
INSERT - Takes(s): 0.0, Count: 20, OPS: 2167.9, Avg(us): 2709, Min(us): 1200, Max(us): 5071, 95th(us): 6000, 99th(us): 6000
***************** properties *****************
"mysql.db"="changefeed_auto_stop_4"
"recordcount"="20"
"requestdistribution"="uniform"
"updateproportion"="0"
"mysql.user"="root"
"insertproportion"="0"
"readallfields"="true"
"mysql.port"="4000"
"readproportion"="0"
"operationcount"="0"
"threadcount"="4"
"dotransactions"="false"
"workload"="core"
"scanproportion"="0"
"mysql.host"="127.0.0.1"
**********************************************
Run finished, takes 11.171942ms
INSERT - Takes(s): 0.0, Count: 20, OPS: 3205.4, Avg(us): 2111, Min(us): 1232, Max(us): 4868, 95th(us): 5000, 99th(us): 5000
[Tue May 7 15:27:42 CST 2024] <<<<<< START cdc server in changefeed_auto_stop case >>>>>>
+ [[ '' == \t\r\u\e ]]
+ set +e
+ get_info_fail_msg='failed to get info:'
+ etcd_info_msg='etcd info'
+ '[' -z '' ']'
+ curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL'
+ GO_FAILPOINTS=
+ [[ no != \n\o ]]
+ cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_auto_stop.1340713409.out server --log-file /tmp/tidb_cdc_test/changefeed_auto_stop/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_auto_stop/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379
+ (( i = 0 ))
+ (( i <= 50 ))
++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL
* About to connect() to 127.0.0.1 port 8301 (#0)
* Trying 127.0.0.1...
* Connection refused
* Failed connect to 127.0.0.1:8301; Connection refused
* Closing connection 0
+ res=
+ echo ''
+ grep -q 'failed to get info:'
+ echo ''
+ grep -q 'etcd info'
+ '[' 0 -eq 50 ']'
+ sleep 3
+ (( i++ ))
+ (( i <= 50 ))
++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL
* About to connect() to 127.0.0.1 port 8301 (#0)
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0)
* Server auth using Basic with user 'ticdc'
> GET /debug/info HTTP/1.1
> Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:8301
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Tue, 07 May 2024 07:27:45 GMT
< Content-Length: 828
< Content-Type: text/plain; charset=utf-8
<
{ [data not shown]
* Connection #0 to host 127.0.0.1 left intact
+ res='
*** owner info ***:
*** processors info ***:
*** etcd info ***:
/tidb/cdc/default/__cdc_meta__/capture/d0912f24-858c-47c2-8a4f-ba25ef3c667c
{"id":"d0912f24-858c-47c2-8a4f-ba25ef3c667c","address":"127.0.0.1:8301","version":"v8.1.0-alpha-26-g299703e05","git-hash":"299703e05932fae47116ddc8fcf22bdcb4e98ea7","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1715066862}
/tidb/cdc/default/__cdc_meta__/meta/meta-version
1
/tidb/cdc/default/__cdc_meta__/owner/22318f51f3385185
d0912f24-858c-47c2-8a4f-ba25ef3c667c
/tidb/cdc/default/default/upstream/7366156010194973418
{"id":7366156010194973418,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}'
+ echo '
*** owner info ***:
*** processors info ***:
*** etcd info ***:
/tidb/cdc/default/__cdc_meta__/capture/d0912f24-858c-47c2-8a4f-ba25ef3c667c
{"id":"d0912f24-858c-47c2-8a4f-ba25ef3c667c","address":"127.0.0.1:8301","version":"v8.1.0-alpha-26-g299703e05","git-hash":"299703e05932fae47116ddc8fcf22bdcb4e98ea7","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1715066862}
/tidb/cdc/default/__cdc_meta__/meta/meta-version
1
/tidb/cdc/default/__cdc_meta__/owner/22318f51f3385185
d0912f24-858c-47c2-8a4f-ba25ef3c667c
/tidb/cdc/default/default/upstream/7366156010194973418
{"id":7366156010194973418,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}'
+ grep -q 'failed to get info:'
+ echo '
*** owner info ***:
*** processors info ***:
*** etcd info ***:
/tidb/cdc/default/__cdc_meta__/capture/d0912f24-858c-47c2-8a4f-ba25ef3c667c
{"id":"d0912f24-858c-47c2-8a4f-ba25ef3c667c","address":"127.0.0.1:8301","version":"v8.1.0-alpha-26-g299703e05","git-hash":"299703e05932fae47116ddc8fcf22bdcb4e98ea7","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1715066862}
/tidb/cdc/default/__cdc_meta__/meta/meta-version
1
/tidb/cdc/default/__cdc_meta__/owner/22318f51f3385185
d0912f24-858c-47c2-8a4f-ba25ef3c667c
/tidb/cdc/default/default/upstream/7366156010194973418
{"id":7366156010194973418,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}'
+ grep -q 'etcd info'
+ break
+ set +x
[Tue May 7 15:27:45 CST 2024] <<<<<< START cdc server in changefeed_auto_stop case >>>>>>
+ [[ '' == \t\r\u\e ]]
+ set +e
+ get_info_fail_msg='failed to get info:'
+ etcd_info_msg='etcd info'
+ '[' -z '' ']'
+ curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL'
+ [[ no != \n\o ]]
+ GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/pipeline/ProcessorSyncResolvedError=1*return(true);github.com/pingcap/tiflow/cdc/processor/ProcessorUpdatePositionDelaying=sleep(1000)'
+ (( i = 0 ))
+ (( i <= 50 ))
+ cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_auto_stop.1347113473.out server --log-file /tmp/tidb_cdc_test/changefeed_auto_stop/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_auto_stop/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379
++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL
* About to connect() to 127.0.0.1 port 8302 (#0)
* Trying 127.0.0.1...
* Connection refused
* Failed connect to 127.0.0.1:8302; Connection refused
* Closing connection 0
+ res=
+ echo ''
+ grep -q 'failed to get info:'
+ echo ''
+ grep -q 'etcd info'
+ '[' 0 -eq 50 ']'
+ sleep 3
+ (( i++ ))
+ (( i <= 50 ))
++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL
* About to connect() to 127.0.0.1 port 8302 (#0)
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0)
* Server auth using Basic with user 'ticdc'
> GET /debug/info HTTP/1.1
> Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:8302
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Tue, 07 May 2024 07:27:48 GMT
< Content-Length: 1297
< Content-Type: text/plain; charset=utf-8
<
{ [data not shown]
* Connection #0 to host 127.0.0.1 left intact
+ res='
*** processors info ***:
*** etcd info ***:
/tidb/cdc/default/__cdc_meta__/capture/0b4d9246-1a05-4485-9007-332da650791f
{"id":"0b4d9246-1a05-4485-9007-332da650791f","address":"127.0.0.1:8302","version":"v8.1.0-alpha-26-g299703e05","git-hash":"299703e05932fae47116ddc8fcf22bdcb4e98ea7","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1715066865}
/tidb/cdc/default/__cdc_meta__/capture/d0912f24-858c-47c2-8a4f-ba25ef3c667c
{"id":"d0912f24-858c-47c2-8a4f-ba25ef3c667c","address":"127.0.0.1:8301","version":"v8.1.0-alpha-26-g299703e05","git-hash":"299703e05932fae47116ddc8fcf22bdcb4e98ea7","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1715066862}
/tidb/cdc/default/__cdc_meta__/meta/meta-version
1
/tidb/cdc/default/__cdc_meta__/owner/22318f51f3385185
d0912f24-858c-47c2-8a4f-ba25ef3c667c
/tidb/cdc/default/__cdc_meta__/owner/22318f51f33851bb
0b4d9246-1a05-4485-9007-332da650791f
/tidb/cdc/default/default/upstream/7366156010194973418
{"id":7366156010194973418,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}'
+ echo '
*** processors info ***:
*** etcd info ***:
/tidb/cdc/default/__cdc_meta__/capture/0b4d9246-1a05-4485-9007-332da650791f
{"id":"0b4d9246-1a05-4485-9007-332da650791f","address":"127.0.0.1:8302","version":"v8.1.0-alpha-26-g299703e05","git-hash":"299703e05932fae47116ddc8fcf22bdcb4e98ea7","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1715066865}
/tidb/cdc/default/__cdc_meta__/capture/d0912f24-858c-47c2-8a4f-ba25ef3c667c
{"id":"d0912f24-858c-47c2-8a4f-ba25ef3c667c","address":"127.0.0.1:8301","version":"v8.1.0-alpha-26-g299703e05","git-hash":"299703e05932fae47116ddc8fcf22bdcb4e98ea7","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1715066862}
/tidb/cdc/default/__cdc_meta__/meta/meta-version
1
/tidb/cdc/default/__cdc_meta__/owner/22318f51f3385185
d0912f24-858c-47c2-8a4f-ba25ef3c667c
/tidb/cdc/default/__cdc_meta__/owner/22318f51f33851bb
0b4d9246-1a05-4485-9007-332da650791f
/tidb/cdc/default/default/upstream/7366156010194973418
{"id":7366156010194973418,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}'
+ grep -q 'failed to get info:'
+ echo '
*** processors info ***:
*** etcd info ***:
/tidb/cdc/default/__cdc_meta__/capture/0b4d9246-1a05-4485-9007-332da650791f
{"id":"0b4d9246-1a05-4485-9007-332da650791f","address":"127.0.0.1:8302","version":"v8.1.0-alpha-26-g299703e05","git-hash":"299703e05932fae47116ddc8fcf22bdcb4e98ea7","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1715066865}
/tidb/cdc/default/__cdc_meta__/capture/d0912f24-858c-47c2-8a4f-ba25ef3c667c
{"id":"d0912f24-858c-47c2-8a4f-ba25ef3c667c","address":"127.0.0.1:8301","version":"v8.1.0-alpha-26-g299703e05","git-hash":"299703e05932fae47116ddc8fcf22bdcb4e98ea7","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1715066862}
/tidb/cdc/default/__cdc_meta__/meta/meta-version
1
/tidb/cdc/default/__cdc_meta__/owner/22318f51f3385185
d0912f24-858c-47c2-8a4f-ba25ef3c667c
/tidb/cdc/default/__cdc_meta__/owner/22318f51f33851bb
0b4d9246-1a05-4485-9007-332da650791f
/tidb/cdc/default/default/upstream/7366156010194973418
{"id":7366156010194973418,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}'
+ grep -q 'etcd info'
+ break
+ set +x
++ date
+ echo '[Tue May 7 15:27:48 CST 2024] <<<<<< START pulsar cluster in changefeed_auto_stop case >>>>>>'
[Tue May 7 15:27:48 CST 2024] <<<<<< START pulsar cluster in changefeed_auto_stop case >>>>>>
+ workdir=/tmp/tidb_cdc_test/changefeed_auto_stop
+ cluster_type=normal
+ cd /tmp/tidb_cdc_test/changefeed_auto_stop
+ DEFAULT_PULSAR_HOME=/usr/local/pulsar
+ pulsar_dir=/usr/local/pulsar
++ cat
+ mtls_conf='
authenticationEnabled=true
authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls
brokerClientTlsEnabled=true
brokerClientTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_auto_stop/ca.cert.pem
brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls
brokerClientAuthenticationParameters={"tlsCertFile":"/tmp/tidb_cdc_test/changefeed_auto_stop/broker_client.cert.pem","tlsKeyFile":"/tmp/tidb_cdc_test/changefeed_auto_stop/broker_client.key-pk8.pem"}
brokerServicePortTls=6651
webServicePortTls=8443
tlsTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_auto_stop/ca.cert.pem
tlsCertificateFilePath=/tmp/tidb_cdc_test/changefeed_auto_stop/server.cert.pem
tlsKeyFilePath=/tmp/tidb_cdc_test/changefeed_auto_stop/server.key-pk8.pem
tlsRequireTrustedClientCertOnConnect=true
tlsAllowInsecureConnection=false
tlsCertRefreshCheckDurationSec=300'
++ cat
+ normal_client_conf='
webServiceUrl=http://localhost:8080/
brokerServiceUrl=pulsar://localhost:6650/'
++ cat
+ mtls_client_conf='
webServiceUrl=https://localhost:8443/
brokerServiceUrl=pulsar+ssl://localhost:6651/
authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls
authParams=tlsCertFile:/tmp/tidb_cdc_test/changefeed_auto_stop/broker_client.cert.pem,tlsKeyFile:/tmp/tidb_cdc_test/changefeed_auto_stop/broker_client.key-pk8.pem
tlsTrustCertsFilePath=/tmp/tidb_cdc_test/changefeed_auto_stop/ca.cert.pem'
++ cat
+ oauth_client_conf='
webServiceUrl=http://localhost:8080/
brokerServiceUrl=pulsar://localhost:6650/
authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2
authParams={"privateKey":"/tmp/tidb_cdc_test/changefeed_auto_stop/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}'
++ cat
+ oauth_conf='
authenticationEnabled=true
authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken
brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2
brokerClientAuthenticationParameters={"privateKey":"file:///tmp/tidb_cdc_test/changefeed_auto_stop/credential.json","audience":"cdc-api-uri","issuerUrl":"http://localhost:9096"}
tokenSecretKey=data:;base64,U0poWDM2X0thcFlTeWJCdEpxMzVseFhfQnJyNExSVVJTa203UW1YSkdteThwVUZXOUVJT2NWUVBzeWt6OS1qag=='
++ cat
+ credential_json='
{
"client_id":"1234",
"client_secret":"e0KVlA2EiBfjoN13olyZd2kv1KL",
"audience":"cdc-api-uri",
"issuer_url":"http://localhost:9096",
"type": "client_credentials"
}'
++ cat
+ cert_server_conf='[ req ]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = dn
[ v3_ext ]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints=CA:FALSE
keyUsage=critical, digitalSignature, keyEncipherment
extendedKeyUsage=serverAuth
subjectAltName=@alt_names
[ dn ]
CN = server
[ alt_names ]
DNS.1 = localhost
IP.1 = 127.0.0.1'
+ echo '
webServiceUrl=http://localhost:8080/
brokerServiceUrl=pulsar://localhost:6650/'
+ cp /usr/local/pulsar/conf/standalone.conf /tmp/tidb_cdc_test/changefeed_auto_stop/pulsar_standalone.conf
+ pulsar_port=6650
+ '[' normal == mtls ']'
+ '[' normal == oauth ']'
+ echo 'no cluster type specified, using default configuration.'
no cluster type specified, using default configuration.
++ date
+ echo '[Tue May 7 15:27:48 CST 2024] <<<<<< START pulsar cluster in normal mode in changefeed_auto_stop case >>>>>>'
[Tue May 7 15:27:48 CST 2024] <<<<<< START pulsar cluster in normal mode in changefeed_auto_stop case >>>>>>
+ echo 'Waiting for pulsar port to be ready...'
Waiting for pulsar port to be ready...
+ i=0
+ nc -z localhost 6650
+ /usr/local/pulsar/bin/pulsar standalone --config /tmp/tidb_cdc_test/changefeed_auto_stop/pulsar_standalone.conf -nfw --metadata-dir /tmp/tidb_cdc_test/changefeed_auto_stop/pulsar-metadata --bookkeeper-dir /tmp/tidb_cdc_test/changefeed_auto_stop/pulsar-bookie
+ i=1
+ '[' 1 -gt 20 ']'
+ sleep 2
+ nc -z localhost 6650
+ i=2
+ '[' 2 -gt 20 ']'
+ sleep 2
+ nc -z localhost 6650
+ i=3
+ '[' 3 -gt 20 ']'
+ sleep 2
+ nc -z localhost 6650
+ echo 'Waiting for pulsar namespace to be ready...'
Waiting for pulsar namespace to be ready...
+ i=0
+ /usr/local/pulsar/bin/pulsar-admin namespaces list public
public/default
++ date
+ echo '[Tue May 7 15:27:57 CST 2024] <<<<<< pulsar is ready >>>>>>'
[Tue May 7 15:27:57 CST 2024] <<<<<< pulsar is ready >>>>>>
[Tue May 7 15:27:58 CST 2024] <<<<<< START Pulsar consumer in changefeed_auto_stop case >>>>>>
check_changefeed_state http://127.0.0.1:2379 b15e4600-a45f-45dc-9a0b-0f1cb30ae1bb normal null
+ endpoints=http://127.0.0.1:2379
+ changefeed_id=b15e4600-a45f-45dc-9a0b-0f1cb30ae1bb
+ expected_state=normal
+ error_msg=null
+ tls_dir=null
+ [[ http://127.0.0.1:2379 =~ https ]]
++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c b15e4600-a45f-45dc-9a0b-0f1cb30ae1bb -s
+ info='{
"upstream_id": 7366156010194973418,
"namespace": "default",
"id": "b15e4600-a45f-45dc-9a0b-0f1cb30ae1bb",
"state": "normal",
"checkpoint_tso": 449594487294394369,
"checkpoint_time": "2024-05-07 15:27:41.322",
"error": null
}'
+ echo '{
"upstream_id": 7366156010194973418,
"namespace": "default",
"id": "b15e4600-a45f-45dc-9a0b-0f1cb30ae1bb",
"state": "normal",
"checkpoint_tso": 449594487294394369,
"checkpoint_time": "2024-05-07 15:27:41.322",
"error": null
}'
{
"upstream_id": 7366156010194973418,
"namespace": "default",
"id": "b15e4600-a45f-45dc-9a0b-0f1cb30ae1bb",
"state": "normal",
"checkpoint_tso": 449594487294394369,
"checkpoint_time": "2024-05-07 15:27:41.322",
"error": null
}
++ echo '{' '"upstream_id":' 7366156010194973418, '"namespace":' '"default",' '"id":' '"b15e4600-a45f-45dc-9a0b-0f1cb30ae1bb",' '"state":' '"normal",' '"checkpoint_tso":' 449594487294394369, '"checkpoint_time":' '"2024-05-07' '15:27:41.322",' '"error":' null '}'
++ jq -r .state
+ state=normal
+ [[ ! normal == \n\o\r\m\a\l ]]
++ echo '{' '"upstream_id":' 7366156010194973418, '"namespace":' '"default",' '"id":' '"b15e4600-a45f-45dc-9a0b-0f1cb30ae1bb",' '"state":' '"normal",' '"checkpoint_tso":' 449594487294394369, '"checkpoint_time":' '"2024-05-07' '15:27:41.322",' '"error":' null '}'
++ jq -r .error.message
+ message=null
+ [[ ! null =~ null ]]
run task successfully
table changefeed_auto_stop_1.usertable not exists for 1-th check, retry later
table changefeed_auto_stop_1.usertable exists
table changefeed_auto_stop_2.usertable not exists for 1-th check, retry later
table changefeed_auto_stop_2.usertable not exists for 2-th check, retry later
table changefeed_auto_stop_2.usertable exists
table changefeed_auto_stop_3.usertable not exists for 1-th check, retry later
table changefeed_auto_stop_3.usertable exists
table changefeed_auto_stop_4.usertable not exists for 1-th check, retry later
table changefeed_auto_stop_4.usertable exists
check diff failed 1-th time, retry later
check diff successfully
wait process cdc.test exit for 1-th time...
wait process cdc.test exit for 2-th time...
wait process cdc.test exit for 3-th time...
cdc.test: no process found
wait process cdc.test exit for 4-th time...
process cdc.test already exit
[Tue May 7 15:28:12 CST 2024] <<<<<< run test case changefeed_auto_stop success! >>>>>>
\033[0;36m<<< Run all test success >>>\033[0m
[Pipeline] }
Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-77/tiflow-cdc already exists)
[Pipeline] // cache
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 723 100 723 0 0 10286 0 --:--:-- --:--:-- --:--:-- 10328
+ synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-07 15:26:33.657","puller_resolved_ts":"2024-05-07 15:26:33.657","last_synced_ts":"2024-05-07 15:26:27.157","now_ts":"2024-05-07 15:28:44.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}'
++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-07' '15:26:33.657","puller_resolved_ts":"2024-05-07' '15:26:33.657","last_synced_ts":"2024-05-07' '15:26:27.157","now_ts":"2024-05-07' '15:28:44.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}'
++ jq .synced
+ status=false
+ '[' false '!=' false ']'
++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-07' '15:26:33.657","puller_resolved_ts":"2024-05-07' '15:26:33.657","last_synced_ts":"2024-05-07' '15:26:27.157","now_ts":"2024-05-07' '15:28:44.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}'
++ jq -r .info
+ info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait'
+ target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait'
+ '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']'
+ cleanup_process cdc.test
wait process cdc.test exit for 1-th time...
wait process cdc.test exit for 2-th time...
wait process cdc.test exit for 3-th time...
cdc.test: no process found
wait process cdc.test exit for 4-th time...
process cdc.test already exit
+ stop_tidb_cluster
+ run_case_with_unavailable_tidb conf/changefeed.toml
+ rm -rf /tmp/tidb_cdc_test/synced_status
+ mkdir -p /tmp/tidb_cdc_test/synced_status
+ start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
The 1 times to try to start tidb cluster...
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
start tidb cluster in /tmp/tidb_cdc_test/synced_status
Starting Upstream PD...
Release Version: v8.1.0-alpha-20-g13eaf8aaa
Edition: Community
Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d
Git Branch: release-8.1
UTC Build Time: 2024-04-30 07:22:55
Starting Downstream PD...
Release Version: v8.1.0-alpha-20-g13eaf8aaa
Edition: Community
Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d
Git Branch: release-8.1
UTC Build Time: 2024-04-30 07:22:55
Verifying upstream PD is started...
Verifying downstream PD is started...
Starting Upstream TiKV...
TiKV
Release Version: 8.1.0
Edition: Community
Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8
Git Commit Branch: release-8.1
UTC Build Time: 2024-04-30 06:15:40
Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27)
Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored
Profile: dist_release
Starting Downstream TiKV...
TiKV
Release Version: 8.1.0
Edition: Community
Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8
Git Commit Branch: release-8.1
UTC Build Time: 2024-04-30 06:15:40
Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27)
Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored
Profile: dist_release
Starting Upstream TiDB...
Release Version: v8.1.0-alpha-106-gc8e6662d10
Edition: Community
Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991
Git Branch: release-8.1
UTC Build Time: 2024-05-06 10:39:26
GoVersion: go1.21.6
Race Enabled: false
Check Table Before Drop: false
Store: unistore
Starting Downstream TiDB...
Release Version: v8.1.0-alpha-106-gc8e6662d10
Edition: Community
Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991
Git Branch: release-8.1
UTC Build Time: 2024-05-06 10:39:26
GoVersion: go1.21.6
Race Enabled: false
Check Table Before Drop: false
Store: unistore
Verifying Upstream TiDB is started...
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
VARIABLE_NAME VARIABLE_VALUE COMMENT
bootstrapped True Bootstrap flag. Do not delete.
tidb_server_version 196 Bootstrap version. Do not delete.
system_tz Asia/Shanghai TiDB Global System Timezone.
new_collation_enabled True If the new collations are enabled. Do not edit it.
ddl_table_version 3 DDL Table Version. Do not delete.
tikv_gc_leader_uuid 63d47d3248c0013 Current GC worker leader UUID. (DO NOT EDIT)
tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-77-t9zbc, pid:24497, start at 2024-05-07 15:29:07.920518499 +0800 CST m=+5.524523032 Host name and pid of current GC leader. (DO NOT EDIT)
tikv_gc_leader_lease 20240507-15:31:07.928 +0800 Current GC worker leader lease. (DO NOT EDIT)
tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used
tikv_gc_enable true Current GC enable status
tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format.
tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format.
tikv_gc_last_run_time 20240507-15:29:07.925 +0800 The time when last GC starts. (DO NOT EDIT)
tikv_gc_safe_point 20240507-15:19:07.925 +0800 All versions after safe point can be accessed. (DO NOT EDIT)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
VARIABLE_NAME VARIABLE_VALUE COMMENT
bootstrapped True Bootstrap flag. Do not delete.
tidb_server_version 196 Bootstrap version. Do not delete.
system_tz Asia/Shanghai TiDB Global System Timezone.
new_collation_enabled True If the new collations are enabled. Do not edit it.
ddl_table_version 3 DDL Table Version. Do not delete.
tikv_gc_leader_uuid 63d47d3248c0013 Current GC worker leader UUID. (DO NOT EDIT)
tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-77-t9zbc, pid:24497, start at 2024-05-07 15:29:07.920518499 +0800 CST m=+5.524523032 Host name and pid of current GC leader. (DO NOT EDIT)
tikv_gc_leader_lease 20240507-15:31:07.928 +0800 Current GC worker leader lease. (DO NOT EDIT)
tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used
tikv_gc_enable true Current GC enable status
tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format.
tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format.
tikv_gc_last_run_time 20240507-15:29:07.925 +0800 The time when last GC starts. (DO NOT EDIT)
tikv_gc_safe_point 20240507-15:19:07.925 +0800 All versions after safe point can be accessed. (DO NOT EDIT)
Verifying Downstream TiDB is started...
VARIABLE_NAME VARIABLE_VALUE COMMENT
bootstrapped True Bootstrap flag. Do not delete.
tidb_server_version 196 Bootstrap version. Do not delete.
system_tz Asia/Shanghai TiDB Global System Timezone.
new_collation_enabled True If the new collations are enabled. Do not edit it.
ddl_table_version 3 DDL Table Version. Do not delete.
tikv_gc_leader_uuid 63d47d326080005 Current GC worker leader UUID. (DO NOT EDIT)
tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-77-t9zbc, pid:24579, start at 2024-05-07 15:29:07.979125598 +0800 CST m=+5.532281977 Host name and pid of current GC leader. (DO NOT EDIT)
tikv_gc_leader_lease 20240507-15:31:07.986 +0800 Current GC worker leader lease. (DO NOT EDIT)
tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used
tikv_gc_enable true Current GC enable status
tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format.
tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format.
tikv_gc_last_run_time 20240507-15:29:07.970 +0800 The time when last GC starts. (DO NOT EDIT)
tikv_gc_safe_point 20240507-15:19:07.970 +0800 All versions after safe point can be accessed. (DO NOT EDIT)
Starting Upstream TiFlash...
TiFlash
Release Version: v8.1.0-alpha-20-geb96e7201
Edition: Community
Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c
Git Branch: HEAD
UTC Build Time: 2024-05-05 06:18:35
Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto
Profile: RELWITHDEBINFO
Compiler: clang++ 13.0.0
Raft Proxy
Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79
Git Commit Branch: HEAD
UTC Build Time: 2024-05-05 06:22:21
Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14)
Storage Engine: tiflash
Prometheus Prefix: tiflash_proxy_
Profile: release
Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored
Verifying Upstream TiFlash is started...
Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log
Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log
arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
+ cd /tmp/tidb_cdc_test/synced_status
++ run_cdc_cli_tso_query 127.0.0.1 2379
+ pd_host=127.0.0.1
+ pd_port=2379
+ is_tls=false
+ '[' false == true ']'
++ run_cdc_cli tso query --pd=http://127.0.0.1:2379
+ cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.25983.out cli tso query --pd=http://127.0.0.1:2379
+ set +x
+ tso='449594511360000001
PASS
coverage: 2.0% of statements in github.com/pingcap/tiflow/...'
+ echo 449594511360000001 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...
+ awk -F ' ' '{print $1}'
+ set +x
+ start_ts=449594511360000001
+ run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test
[Tue May 7 15:29:14 CST 2024] <<<<<< START cdc server in synced_status case >>>>>>
+ [[ '' == \t\r\u\e ]]
+ set +e
+ get_info_fail_msg='failed to get info:'
+ etcd_info_msg='etcd info'
+ '[' -z '' ']'
+ curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL'
+ GO_FAILPOINTS=
+ [[ no != \n\o ]]
+ cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2602026022.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default
+ (( i = 0 ))
+ (( i <= 50 ))
++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL
* About to connect() to 127.0.0.1 port 8300 (#0)
* Trying 127.0.0.1...
* Connection refused
* Failed connect to 127.0.0.1:8300; Connection refused
* Closing connection 0
+ res=
+ echo ''
+ grep -q 'failed to get info:'
+ echo ''
+ grep -q 'etcd info'
+ '[' 0 -eq 50 ']'
+ sleep 3
+ (( i++ ))
+ (( i <= 50 ))
++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL
* About to connect() to 127.0.0.1 port 8300 (#0)
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0)
* Server auth using Basic with user 'ticdc'
> GET /debug/info HTTP/1.1
> Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:8300
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Tue, 07 May 2024 07:29:17 GMT
< Content-Length: 828
< Content-Type: text/plain; charset=utf-8
<
{ [data not shown]
* Connection #0 to host 127.0.0.1 left intact
+ res='
*** owner info ***:
*** processors info ***:
*** etcd info ***:
/tidb/cdc/default/__cdc_meta__/capture/98a50de7-4841-46fb-8597-72a64822a9bf
{"id":"98a50de7-4841-46fb-8597-72a64822a9bf","address":"127.0.0.1:8300","version":"v8.1.0-alpha-26-g299703e05","git-hash":"299703e05932fae47116ddc8fcf22bdcb4e98ea7","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1715066954}
/tidb/cdc/default/__cdc_meta__/meta/meta-version
1
/tidb/cdc/default/__cdc_meta__/owner/22318f51f4a1a93c
98a50de7-4841-46fb-8597-72a64822a9bf
/tidb/cdc/default/default/upstream/7366156411075073671
{"id":7366156411075073671,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}'
+ echo '
*** owner info ***:
*** processors info ***:
*** etcd info ***:
/tidb/cdc/default/__cdc_meta__/capture/98a50de7-4841-46fb-8597-72a64822a9bf
{"id":"98a50de7-4841-46fb-8597-72a64822a9bf","address":"127.0.0.1:8300","version":"v8.1.0-alpha-26-g299703e05","git-hash":"299703e05932fae47116ddc8fcf22bdcb4e98ea7","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1715066954}
/tidb/cdc/default/__cdc_meta__/meta/meta-version
1
/tidb/cdc/default/__cdc_meta__/owner/22318f51f4a1a93c
98a50de7-4841-46fb-8597-72a64822a9bf
/tidb/cdc/default/default/upstream/7366156411075073671
{"id":7366156411075073671,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}'
+ grep -q 'failed to get info:'
+ echo '
*** owner info ***:
*** processors info ***:
*** etcd info ***:
/tidb/cdc/default/__cdc_meta__/capture/98a50de7-4841-46fb-8597-72a64822a9bf
{"id":"98a50de7-4841-46fb-8597-72a64822a9bf","address":"127.0.0.1:8300","version":"v8.1.0-alpha-26-g299703e05","git-hash":"299703e05932fae47116ddc8fcf22bdcb4e98ea7","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1715066954}
/tidb/cdc/default/__cdc_meta__/meta/meta-version
1
/tidb/cdc/default/__cdc_meta__/owner/22318f51f4a1a93c
98a50de7-4841-46fb-8597-72a64822a9bf
/tidb/cdc/default/default/upstream/7366156411075073671
{"id":7366156411075073671,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}'
+ grep -q 'etcd info'
+ break
+ set +x
+ config_path=conf/changefeed.toml
+ SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1'
+ run_cdc_cli changefeed create --start-ts=449594511360000001 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml
+ cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.26079.out cli changefeed create --start-ts=449594511360000001 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml
Create changefeed successfully!
ID: test-1
Info: {"upstream_id":7366156411075073671,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-07T15:29:18.17143196+08:00","start_ts":449594511360000001,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v8.1.0-alpha-26-g299703e05","resolved_ts":449594511360000001,"checkpoint_ts":449594511360000001,"checkpoint_time":"2024-05-07 15:29:13.125"}
PASS
coverage: 2.6% of statements in github.com/pingcap/tiflow/...
+ set +x
+ run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);'
+ check_table_exists test.t1 127.0.0.1 3306
table test.t1 not exists for 1-th check, retry later
table test.t1 exists
+ sleep 5
+ kill_tidb
++ ps aux
++ grep tidb-server
++ grep /tmp/tidb_cdc_test/synced_status
+ info='jenkins 24497 14.0 0.0 2515004 260968 ? Sl 15:29 0:03 tidb-server -P 4000 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1715066942389791086.toml --store tikv --path 127.0.0.1:2379 --status=10080 --log-file /tmp/tidb_cdc_test/synced_status/tidb.log
jenkins 24501 4.1 0.0 2512632 192760 ? Sl 15:29 0:00 tidb-server -P 4001 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1715066942392920059.toml --store tikv --path 127.0.0.1:2379 --status=10081 --log-file /tmp/tidb_cdc_test/synced_status/tidb_other.log
jenkins 24579 14.1 0.0 2820772 278112 ? Sl 15:29 0:03 tidb-server -P 3306 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1715066942440375209.toml --store tikv --path 127.0.0.1:2479 --status=20080 --log-file /tmp/tidb_cdc_test/synced_status/tidb_down.log'
++ ps aux
++ grep tidb-server
++ grep /tmp/tidb_cdc_test/synced_status
++ awk '{print $2}'
++ xargs kill -9
++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 243 100 243 0 0 2298 0 --:--:-- --:--:-- --:--:-- 2314
+ synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-07 15:29:26.425","puller_resolved_ts":"2024-05-07 15:29:19.775","last_synced_ts":"2024-05-07 15:29:19.825","now_ts":"2024-05-07 15:29:26.000","info":"The data syncing is not finished, please wait"}'
++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-07' '15:29:26.425","puller_resolved_ts":"2024-05-07' '15:29:19.775","last_synced_ts":"2024-05-07' '15:29:19.825","now_ts":"2024-05-07' '15:29:26.000","info":"The' data syncing is not finished, please 'wait"}'
++ jq .synced
+ status=false
+ '[' false '!=' false ']'
++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-07' '15:29:26.425","puller_resolved_ts":"2024-05-07' '15:29:19.775","last_synced_ts":"2024-05-07' '15:29:19.825","now_ts":"2024-05-07' '15:29:26.000","info":"The' data syncing is not finished, please 'wait"}'
++ jq -r .info
+ info='The data syncing is not finished, please wait'
+ target_message='The data syncing is not finished, please wait'
+ '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']'
+ sleep 130
++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 723 100 723 0 0 9627 0 --:--:-- --:--:-- --:--:-- 9513
100 723 100 723 0 0 9613 0 --:--:-- --:--:-- --:--:-- 9513
+ synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-07 15:27:19.702","puller_resolved_ts":"2024-05-07 15:27:19.702","last_synced_ts":"2024-05-07 15:27:13.002","now_ts":"2024-05-07 15:29:30.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}'
++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-07' '15:27:19.702","puller_resolved_ts":"2024-05-07' '15:27:19.702","last_synced_ts":"2024-05-07' '15:27:13.002","now_ts":"2024-05-07' '15:29:30.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}'
++ jq .synced
+ status=false
+ '[' false '!=' false ']'
++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-07' '15:27:19.702","puller_resolved_ts":"2024-05-07' '15:27:19.702","last_synced_ts":"2024-05-07' '15:27:13.002","now_ts":"2024-05-07' '15:29:30.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}'
++ jq -r .info
+ info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait'
+ target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait'
+ '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']'
+ cleanup_process cdc.test
wait process cdc.test exit for 1-th time...
wait process cdc.test exit for 2-th time...
wait process cdc.test exit for 3-th time...
cdc.test: no process found
wait process cdc.test exit for 4-th time...
process cdc.test already exit
+ stop_tidb_cluster
+ run_case_with_unavailable_tidb conf/changefeed-redo.toml
+ rm -rf /tmp/tidb_cdc_test/synced_status_with_redo
+ mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo
+ start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
The 1 times to try to start tidb cluster...
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo
Starting Upstream PD...
Release Version: v8.1.0-alpha-20-g13eaf8aaa
Edition: Community
Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d
Git Branch: release-8.1
UTC Build Time: 2024-04-30 07:22:55
Starting Downstream PD...
Release Version: v8.1.0-alpha-20-g13eaf8aaa
Edition: Community
Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d
Git Branch: release-8.1
UTC Build Time: 2024-04-30 07:22:55
Verifying upstream PD is started...
Verifying downstream PD is started...
Starting Upstream TiKV...
TiKV
Release Version: 8.1.0
Edition: Community
Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8
Git Commit Branch: release-8.1
UTC Build Time: 2024-04-30 06:15:40
Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27)
Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored
Profile: dist_release
Starting Downstream TiKV...
TiKV
Release Version: 8.1.0
Edition: Community
Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8
Git Commit Branch: release-8.1
UTC Build Time: 2024-04-30 06:15:40
Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27)
Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored
Profile: dist_release
Starting Upstream TiDB...
Release Version: v8.1.0-alpha-106-gc8e6662d10
Edition: Community
Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991
Git Branch: release-8.1
UTC Build Time: 2024-05-06 10:39:26
GoVersion: go1.21.6
Race Enabled: false
Check Table Before Drop: false
Store: unistore
Starting Downstream TiDB...
Release Version: v8.1.0-alpha-106-gc8e6662d10
Edition: Community
Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991
Git Branch: release-8.1
UTC Build Time: 2024-05-06 10:39:26
GoVersion: go1.21.6
Race Enabled: false
Check Table Before Drop: false
Store: unistore
Verifying Upstream TiDB is started...
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
VARIABLE_NAME VARIABLE_VALUE COMMENT
bootstrapped True Bootstrap flag. Do not delete.
tidb_server_version 196 Bootstrap version. Do not delete.
system_tz Asia/Shanghai TiDB Global System Timezone.
new_collation_enabled True If the new collations are enabled. Do not edit it.
ddl_table_version 3 DDL Table Version. Do not delete.
tikv_gc_leader_uuid 63d47d607140012 Current GC worker leader UUID. (DO NOT EDIT)
tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-77-16j3h, pid:16897, start at 2024-05-07 15:29:55.170869639 +0800 CST m=+5.285853953 Host name and pid of current GC leader. (DO NOT EDIT)
tikv_gc_leader_lease 20240507-15:31:55.177 +0800 Current GC worker leader lease. (DO NOT EDIT)
tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used
tikv_gc_enable true Current GC enable status
tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format.
tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format.
tikv_gc_last_run_time 20240507-15:29:55.141 +0800 The time when last GC starts. (DO NOT EDIT)
tikv_gc_safe_point 20240507-15:19:55.141 +0800 All versions after safe point can be accessed. (DO NOT EDIT)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
VARIABLE_NAME VARIABLE_VALUE COMMENT
bootstrapped True Bootstrap flag. Do not delete.
tidb_server_version 196 Bootstrap version. Do not delete.
system_tz Asia/Shanghai TiDB Global System Timezone.
new_collation_enabled True If the new collations are enabled. Do not edit it.
ddl_table_version 3 DDL Table Version. Do not delete.
tikv_gc_leader_uuid 63d47d607140012 Current GC worker leader UUID. (DO NOT EDIT)
tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-77-16j3h, pid:16897, start at 2024-05-07 15:29:55.170869639 +0800 CST m=+5.285853953 Host name and pid of current GC leader. (DO NOT EDIT)
tikv_gc_leader_lease 20240507-15:31:55.177 +0800 Current GC worker leader lease. (DO NOT EDIT)
tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used
tikv_gc_enable true Current GC enable status
tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format.
tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format.
tikv_gc_last_run_time 20240507-15:29:55.141 +0800 The time when last GC starts. (DO NOT EDIT)
tikv_gc_safe_point 20240507-15:19:55.141 +0800 All versions after safe point can be accessed. (DO NOT EDIT)
Verifying Downstream TiDB is started...
VARIABLE_NAME VARIABLE_VALUE COMMENT
bootstrapped True Bootstrap flag. Do not delete.
tidb_server_version 196 Bootstrap version. Do not delete.
system_tz Asia/Shanghai TiDB Global System Timezone.
new_collation_enabled True If the new collations are enabled. Do not edit it.
ddl_table_version 3 DDL Table Version. Do not delete.
tikv_gc_leader_uuid 63d47d60890000f Current GC worker leader UUID. (DO NOT EDIT)
tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-77-16j3h, pid:16977, start at 2024-05-07 15:29:55.253118476 +0800 CST m=+5.315794236 Host name and pid of current GC leader. (DO NOT EDIT)
tikv_gc_leader_lease 20240507-15:31:55.260 +0800 Current GC worker leader lease. (DO NOT EDIT)
tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used
tikv_gc_enable true Current GC enable status
tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format.
tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format.
tikv_gc_last_run_time 20240507-15:29:55.236 +0800 The time when last GC starts. (DO NOT EDIT)
tikv_gc_safe_point 20240507-15:19:55.236 +0800 All versions after safe point can be accessed. (DO NOT EDIT)
Starting Upstream TiFlash...
TiFlash
Release Version: v8.1.0-alpha-20-geb96e7201
Edition: Community
Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c
Git Branch: HEAD
UTC Build Time: 2024-05-05 06:18:35
Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto
Profile: RELWITHDEBINFO
Compiler: clang++ 13.0.0
Raft Proxy
Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79
Git Commit Branch: HEAD
UTC Build Time: 2024-05-05 06:22:21
Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14)
Storage Engine: tiflash
Prometheus Prefix: tiflash_proxy_
Profile: release
Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored
Verifying Upstream TiFlash is started...
Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log
Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log
arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
+ cd /tmp/tidb_cdc_test/synced_status_with_redo
++ run_cdc_cli_tso_query 127.0.0.1 2379
+ pd_host=127.0.0.1
+ pd_port=2379
+ is_tls=false
+ '[' false == true ']'
++ run_cdc_cli tso query --pd=http://127.0.0.1:2379
+ cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.18305.out cli tso query --pd=http://127.0.0.1:2379
+ set +x
+ tso='449594523789557761
PASS
coverage: 2.0% of statements in github.com/pingcap/tiflow/...'
+ echo 449594523789557761 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...
+ awk -F ' ' '{print $1}'
+ set +x
+ start_ts=449594523789557761
+ run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test
[Tue May 7 15:30:02 CST 2024] <<<<<< START cdc server in synced_status_with_redo case >>>>>>
+ [[ '' == \t\r\u\e ]]
+ set +e
+ get_info_fail_msg='failed to get info:'
+ etcd_info_msg='etcd info'
+ '[' -z '' ']'
+ curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL'
+ [[ no != \n\o ]]
+ GO_FAILPOINTS=
+ (( i = 0 ))
+ (( i <= 50 ))
+ cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.1833818340.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default
++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL
* About to connect() to 127.0.0.1 port 8300 (#0)
* Trying 127.0.0.1...
* Connection refused
* Failed connect to 127.0.0.1:8300; Connection refused
* Closing connection 0
+ res=
+ echo ''
+ grep -q 'failed to get info:'
+ echo ''
+ grep -q 'etcd info'
+ '[' 0 -eq 50 ']'
+ sleep 3
+ (( i++ ))
+ (( i <= 50 ))
++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL
* About to connect() to 127.0.0.1 port 8300 (#0)
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0)
* Server auth using Basic with user 'ticdc'
> GET /debug/info HTTP/1.1
> Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:8300
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Tue, 07 May 2024 07:30:05 GMT
< Content-Length: 828
< Content-Type: text/plain; charset=utf-8
<
{ [data not shown]
* Connection #0 to host 127.0.0.1 left intact
+ res='
*** owner info ***:
*** processors info ***:
*** etcd info ***:
/tidb/cdc/default/__cdc_meta__/capture/eca91d8d-5de9-4722-b263-65c06678f3e5
{"id":"eca91d8d-5de9-4722-b263-65c06678f3e5","address":"127.0.0.1:8300","version":"v8.1.0-alpha-26-g299703e05","git-hash":"299703e05932fae47116ddc8fcf22bdcb4e98ea7","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1715067002}
/tidb/cdc/default/__cdc_meta__/meta/meta-version
1
/tidb/cdc/default/__cdc_meta__/owner/22318f51f5556b3b
eca91d8d-5de9-4722-b263-65c06678f3e5
/tidb/cdc/default/default/upstream/7366156616336936772
{"id":7366156616336936772,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}'
+ echo '
*** owner info ***:
*** processors info ***:
*** etcd info ***:
/tidb/cdc/default/__cdc_meta__/capture/eca91d8d-5de9-4722-b263-65c06678f3e5
{"id":"eca91d8d-5de9-4722-b263-65c06678f3e5","address":"127.0.0.1:8300","version":"v8.1.0-alpha-26-g299703e05","git-hash":"299703e05932fae47116ddc8fcf22bdcb4e98ea7","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1715067002}
/tidb/cdc/default/__cdc_meta__/meta/meta-version
1
/tidb/cdc/default/__cdc_meta__/owner/22318f51f5556b3b
eca91d8d-5de9-4722-b263-65c06678f3e5
/tidb/cdc/default/default/upstream/7366156616336936772
{"id":7366156616336936772,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}'
+ grep -q 'failed to get info:'
+ echo '
*** owner info ***:
*** processors info ***:
*** etcd info ***:
/tidb/cdc/default/__cdc_meta__/capture/eca91d8d-5de9-4722-b263-65c06678f3e5
{"id":"eca91d8d-5de9-4722-b263-65c06678f3e5","address":"127.0.0.1:8300","version":"v8.1.0-alpha-26-g299703e05","git-hash":"299703e05932fae47116ddc8fcf22bdcb4e98ea7","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1715067002}
/tidb/cdc/default/__cdc_meta__/meta/meta-version
1
/tidb/cdc/default/__cdc_meta__/owner/22318f51f5556b3b
eca91d8d-5de9-4722-b263-65c06678f3e5
/tidb/cdc/default/default/upstream/7366156616336936772
{"id":7366156616336936772,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}'
+ grep -q 'etcd info'
+ break
+ set +x
+ config_path=conf/changefeed-redo.toml
+ SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1'
+ run_cdc_cli changefeed create --start-ts=449594523789557761 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml
+ cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.18393.out cli changefeed create --start-ts=449594523789557761 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml
Create changefeed successfully!
ID: test-1
Info: {"upstream_id":7366156616336936772,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-07T15:30:05.529905006+08:00","start_ts":449594523789557761,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v8.1.0-alpha-26-g299703e05","resolved_ts":449594523789557761,"checkpoint_ts":449594523789557761,"checkpoint_time":"2024-05-07 15:30:00.540"}
PASS
coverage: 2.7% of statements in github.com/pingcap/tiflow/...
+ set +x
+ run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);'
+ check_table_exists test.t1 127.0.0.1 3306
table test.t1 not exists for 1-th check, retry later
table test.t1 exists
+ sleep 5
+ kill_tidb
++ ps aux
++ grep tidb-server
++ grep /tmp/tidb_cdc_test/synced_status_with_redo
+ info='jenkins 16897 13.0 0.0 2671556 236872 ? Sl 15:29 0:03 tidb-server -P 4000 -config /tmp/tidb_cdc_test/synced_status_with_redo/tidb-config-1715066989878179601.toml --store tikv --path 127.0.0.1:2379 --status=10080 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tidb.log
jenkins 16901 3.8 0.0 2364908 193320 ? Sl 15:29 0:00 tidb-server -P 4001 -config /tmp/tidb_cdc_test/synced_status_with_redo/tidb-config-1715066989881320734.toml --store tikv --path 127.0.0.1:2379 --status=10081 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tidb_other.log
jenkins 16977 13.3 0.0 2794496 259184 ? Sl 15:29 0:03 tidb-server -P 3306 -config /tmp/tidb_cdc_test/synced_status_with_redo/tidb-config-1715066989929841068.toml --store tikv --path 127.0.0.1:2479 --status=20080 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tidb_down.log'
++ ps aux
++ grep tidb-server
++ grep /tmp/tidb_cdc_test/synced_status_with_redo
++ awk '{print $2}'
++ xargs kill -9
++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 243 100 243 0 0 12815 0 --:--:-- --:--:-- --:--:-- 13500
+ synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-07 15:30:12.941","puller_resolved_ts":"2024-05-07 15:30:07.041","last_synced_ts":"2024-05-07 15:30:07.141","now_ts":"2024-05-07 15:30:14.000","info":"The data syncing is not finished, please wait"}'
++ jq .synced
++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-07' '15:30:12.941","puller_resolved_ts":"2024-05-07' '15:30:07.041","last_synced_ts":"2024-05-07' '15:30:07.141","now_ts":"2024-05-07' '15:30:14.000","info":"The' data syncing is not finished, please 'wait"}'
+ status=false
+ '[' false '!=' false ']'
++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-07' '15:30:12.941","puller_resolved_ts":"2024-05-07' '15:30:07.041","last_synced_ts":"2024-05-07' '15:30:07.141","now_ts":"2024-05-07' '15:30:14.000","info":"The' data syncing is not finished, please 'wait"}'
++ jq -r .info
+ info='The data syncing is not finished, please wait'
+ target_message='The data syncing is not finished, please wait'
+ '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']'
+ sleep 130
++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 221 100 221 0 0 2896 0 --:--:-- --:--:-- --:--:-- 2907
+ synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-07 15:31:36.574","puller_resolved_ts":"2024-05-07 15:31:29.575","last_synced_ts":"2024-05-07 15:29:19.825","now_ts":"2024-05-07 15:31:37.000","info":"Data syncing is finished"}'
++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-07' '15:31:36.574","puller_resolved_ts":"2024-05-07' '15:31:29.575","last_synced_ts":"2024-05-07' '15:29:19.825","now_ts":"2024-05-07' '15:31:37.000","info":"Data' syncing is 'finished"}'
++ jq .synced
+ status=true
+ '[' true '!=' true ']'
++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-07' '15:31:36.574","puller_resolved_ts":"2024-05-07' '15:31:29.575","last_synced_ts":"2024-05-07' '15:29:19.825","now_ts":"2024-05-07' '15:31:37.000","info":"Data' syncing is 'finished"}'
++ jq -r .info
+ info='Data syncing is finished'
+ target_message='Data syncing is finished'
+ '[' 'Data syncing is finished' '!=' 'Data syncing is finished' ']'
+ cleanup_process cdc.test
wait process cdc.test exit for 1-th time...
wait process cdc.test exit for 2-th time...
wait process cdc.test exit for 3-th time...
cdc.test: no process found
wait process cdc.test exit for 4-th time...
process cdc.test already exit
+ stop_tidb_cluster
+ run_case_with_failpoint conf/changefeed.toml
+ rm -rf /tmp/tidb_cdc_test/synced_status
+ mkdir -p /tmp/tidb_cdc_test/synced_status
+ start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
The 1 times to try to start tidb cluster...
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
start tidb cluster in /tmp/tidb_cdc_test/synced_status
Starting Upstream PD...
Release Version: v8.1.0-alpha-20-g13eaf8aaa
Edition: Community
Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d
Git Branch: release-8.1
UTC Build Time: 2024-04-30 07:22:55
Starting Downstream PD...
Release Version: v8.1.0-alpha-20-g13eaf8aaa
Edition: Community
Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d
Git Branch: release-8.1
UTC Build Time: 2024-04-30 07:22:55
Verifying upstream PD is started...
Verifying downstream PD is started...
Starting Upstream TiKV...
TiKV
Release Version: 8.1.0
Edition: Community
Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8
Git Commit Branch: release-8.1
UTC Build Time: 2024-04-30 06:15:40
Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27)
Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored
Profile: dist_release
Starting Downstream TiKV...
TiKV
Release Version: 8.1.0
Edition: Community
Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8
Git Commit Branch: release-8.1
UTC Build Time: 2024-04-30 06:15:40
Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27)
Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored
Profile: dist_release
Starting Upstream TiDB...
Release Version: v8.1.0-alpha-106-gc8e6662d10
Edition: Community
Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991
Git Branch: release-8.1
UTC Build Time: 2024-05-06 10:39:26
GoVersion: go1.21.6
Race Enabled: false
Check Table Before Drop: false
Store: unistore
Starting Downstream TiDB...
Release Version: v8.1.0-alpha-106-gc8e6662d10
Edition: Community
Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991
Git Branch: release-8.1
UTC Build Time: 2024-05-06 10:39:26
GoVersion: go1.21.6
Race Enabled: false
Check Table Before Drop: false
Store: unistore
Verifying Upstream TiDB is started...
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
VARIABLE_NAME VARIABLE_VALUE COMMENT
bootstrapped True Bootstrap flag. Do not delete.
tidb_server_version 196 Bootstrap version. Do not delete.
system_tz Asia/Shanghai TiDB Global System Timezone.
new_collation_enabled True If the new collations are enabled. Do not edit it.
ddl_table_version 3 DDL Table Version. Do not delete.
tikv_gc_leader_uuid 63d47ddc9f4000e Current GC worker leader UUID. (DO NOT EDIT)
tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-77-t9zbc, pid:27259, start at 2024-05-07 15:32:02.316266809 +0800 CST m=+5.548488136 Host name and pid of current GC leader. (DO NOT EDIT)
tikv_gc_leader_lease 20240507-15:34:02.325 +0800 Current GC worker leader lease. (DO NOT EDIT)
tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used
tikv_gc_enable true Current GC enable status
tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format.
tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format.
tikv_gc_last_run_time 20240507-15:32:02.301 +0800 The time when last GC starts. (DO NOT EDIT)
tikv_gc_safe_point 20240507-15:22:02.301 +0800 All versions after safe point can be accessed. (DO NOT EDIT)
VARIABLE_NAME VARIABLE_VALUE COMMENT
bootstrapped True Bootstrap flag. Do not delete.
tidb_server_version 196 Bootstrap version. Do not delete.
system_tz Asia/Shanghai TiDB Global System Timezone.
new_collation_enabled True If the new collations are enabled. Do not edit it.
ddl_table_version 3 DDL Table Version. Do not delete.
tikv_gc_leader_uuid 63d47ddc9f4000e Current GC worker leader UUID. (DO NOT EDIT)
tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-77-t9zbc, pid:27259, start at 2024-05-07 15:32:02.316266809 +0800 CST m=+5.548488136 Host name and pid of current GC leader. (DO NOT EDIT)
tikv_gc_leader_lease 20240507-15:34:02.325 +0800 Current GC worker leader lease. (DO NOT EDIT)
tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used
tikv_gc_enable true Current GC enable status
tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format.
tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format.
tikv_gc_last_run_time 20240507-15:32:02.301 +0800 The time when last GC starts. (DO NOT EDIT)
tikv_gc_safe_point 20240507-15:22:02.301 +0800 All versions after safe point can be accessed. (DO NOT EDIT)
Verifying Downstream TiDB is started...
VARIABLE_NAME VARIABLE_VALUE COMMENT
bootstrapped True Bootstrap flag. Do not delete.
tidb_server_version 196 Bootstrap version. Do not delete.
system_tz Asia/Shanghai TiDB Global System Timezone.
new_collation_enabled True If the new collations are enabled. Do not edit it.
ddl_table_version 3 DDL Table Version. Do not delete.
tikv_gc_leader_uuid 63d47ddca680015 Current GC worker leader UUID. (DO NOT EDIT)
tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-77-t9zbc, pid:27333, start at 2024-05-07 15:32:02.378275145 +0800 CST m=+5.551723371 Host name and pid of current GC leader. (DO NOT EDIT)
tikv_gc_leader_lease 20240507-15:34:02.392 +0800 Current GC worker leader lease. (DO NOT EDIT)
tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used
tikv_gc_enable true Current GC enable status
tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format.
tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format.
tikv_gc_last_run_time 20240507-15:32:02.380 +0800 The time when last GC starts. (DO NOT EDIT)
tikv_gc_safe_point 20240507-15:22:02.380 +0800 All versions after safe point can be accessed. (DO NOT EDIT)
Starting Upstream TiFlash...
TiFlash
Release Version: v8.1.0-alpha-20-geb96e7201
Edition: Community
Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c
Git Branch: HEAD
UTC Build Time: 2024-05-05 06:18:35
Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto
Profile: RELWITHDEBINFO
Compiler: clang++ 13.0.0
Raft Proxy
Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79
Git Commit Branch: HEAD
UTC Build Time: 2024-05-05 06:22:21
Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14)
Storage Engine: tiflash
Prometheus Prefix: tiflash_proxy_
Profile: release
Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored
Verifying Upstream TiFlash is started...
Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log
Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log
arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
+ cd /tmp/tidb_cdc_test/synced_status
+ export 'GO_FAILPOINTS=github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)'
+ GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)'
++ run_cdc_cli_tso_query 127.0.0.1 2379
+ pd_host=127.0.0.1
+ pd_port=2379
+ is_tls=false
+ '[' false == true ']'
++ run_cdc_cli tso query --pd=http://127.0.0.1:2379
+ cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.28649.out cli tso query --pd=http://127.0.0.1:2379
+ set +x
+ tso='449594557058252801
PASS
coverage: 2.0% of statements in github.com/pingcap/tiflow/...'
+ echo 449594557058252801 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...
+ awk -F ' ' '{print $1}'
+ set +x
+ start_ts=449594557058252801
+ run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test
[Tue May 7 15:32:08 CST 2024] <<<<<< START cdc server in synced_status case >>>>>>
+ [[ '' == \t\r\u\e ]]
+ set +e
+ get_info_fail_msg='failed to get info:'
+ etcd_info_msg='etcd info'
+ '[' -z '' ']'
+ curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL'
+ GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)'
+ [[ no != \n\o ]]
+ cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2868428686.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default
+ (( i = 0 ))
+ (( i <= 50 ))
++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL
* About to connect() to 127.0.0.1 port 8300 (#0)
* Trying 127.0.0.1...
* Connection refused
* Failed connect to 127.0.0.1:8300; Connection refused
* Closing connection 0
+ res=
+ echo ''
+ grep -q 'failed to get info:'
+ echo ''
+ grep -q 'etcd info'
+ '[' 0 -eq 50 ']'
+ sleep 3
+ (( i++ ))
+ (( i <= 50 ))
++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL
* About to connect() to 127.0.0.1 port 8300 (#0)
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0)
* Server auth using Basic with user 'ticdc'
> GET /debug/info HTTP/1.1
> Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:8300
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Tue, 07 May 2024 07:32:12 GMT
< Content-Length: 828
< Content-Type: text/plain; charset=utf-8
<
{ [data not shown]
* Connection #0 to host 127.0.0.1 left intact
+ res='
*** owner info ***:
*** processors info ***:
*** etcd info ***:
/tidb/cdc/default/__cdc_meta__/capture/6f0fc03d-5d90-4e6d-9e62-d881e3435721
{"id":"6f0fc03d-5d90-4e6d-9e62-d881e3435721","address":"127.0.0.1:8300","version":"v8.1.0-alpha-26-g299703e05","git-hash":"299703e05932fae47116ddc8fcf22bdcb4e98ea7","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1715067129}
/tidb/cdc/default/__cdc_meta__/meta/meta-version
1
/tidb/cdc/default/__cdc_meta__/owner/22318f51f744f43c
6f0fc03d-5d90-4e6d-9e62-d881e3435721
/tidb/cdc/default/default/upstream/7366157158332466338
{"id":7366157158332466338,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}'
+ echo '
*** owner info ***:
*** processors info ***:
*** etcd info ***:
/tidb/cdc/default/__cdc_meta__/capture/6f0fc03d-5d90-4e6d-9e62-d881e3435721
{"id":"6f0fc03d-5d90-4e6d-9e62-d881e3435721","address":"127.0.0.1:8300","version":"v8.1.0-alpha-26-g299703e05","git-hash":"299703e05932fae47116ddc8fcf22bdcb4e98ea7","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1715067129}
/tidb/cdc/default/__cdc_meta__/meta/meta-version
1
/tidb/cdc/default/__cdc_meta__/owner/22318f51f744f43c
6f0fc03d-5d90-4e6d-9e62-d881e3435721
/tidb/cdc/default/default/upstream/7366157158332466338
{"id":7366157158332466338,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}'
+ grep -q 'failed to get info:'
+ echo '
*** owner info ***:
*** processors info ***:
*** etcd info ***:
/tidb/cdc/default/__cdc_meta__/capture/6f0fc03d-5d90-4e6d-9e62-d881e3435721
{"id":"6f0fc03d-5d90-4e6d-9e62-d881e3435721","address":"127.0.0.1:8300","version":"v8.1.0-alpha-26-g299703e05","git-hash":"299703e05932fae47116ddc8fcf22bdcb4e98ea7","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1715067129}
/tidb/cdc/default/__cdc_meta__/meta/meta-version
1
/tidb/cdc/default/__cdc_meta__/owner/22318f51f744f43c
6f0fc03d-5d90-4e6d-9e62-d881e3435721
/tidb/cdc/default/default/upstream/7366157158332466338
{"id":7366157158332466338,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}'
+ grep -q 'etcd info'
+ break
+ set +x
+ config_path=conf/changefeed.toml
+ SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1'
+ run_cdc_cli changefeed create --start-ts=449594557058252801 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml
+ cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.28745.out cli changefeed create --start-ts=449594557058252801 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml
Create changefeed successfully!
ID: test-1
Info: {"upstream_id":7366157158332466338,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-07T15:32:12.599764313+08:00","start_ts":449594557058252801,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v8.1.0-alpha-26-g299703e05","resolved_ts":449594557058252801,"checkpoint_ts":449594557058252801,"checkpoint_time":"2024-05-07 15:32:07.450"}
PASS
coverage: 2.6% of statements in github.com/pingcap/tiflow/...
+ set +x
+ sleep 20
++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 221 100 221 0 0 2644 0 --:--:-- --:--:-- --:--:-- 2630
100 221 100 221 0 0 2641 0 --:--:-- --:--:-- --:--:-- 2630
+ synced_status='{"synced":true,"sink_checkpoint_ts":"2024-05-07 15:32:23.041","puller_resolved_ts":"2024-05-07 15:32:17.041","last_synced_ts":"2024-05-07 15:30:07.141","now_ts":"2024-05-07 15:32:24.000","info":"Data syncing is finished"}'
++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-07' '15:32:23.041","puller_resolved_ts":"2024-05-07' '15:32:17.041","last_synced_ts":"2024-05-07' '15:30:07.141","now_ts":"2024-05-07' '15:32:24.000","info":"Data' syncing is 'finished"}'
++ jq .synced
+ status=true
+ '[' true '!=' true ']'
++ echo '{"synced":true,"sink_checkpoint_ts":"2024-05-07' '15:32:23.041","puller_resolved_ts":"2024-05-07' '15:32:17.041","last_synced_ts":"2024-05-07' '15:30:07.141","now_ts":"2024-05-07' '15:32:24.000","info":"Data' syncing is 'finished"}'
++ jq -r .info
+ info='Data syncing is finished'
+ target_message='Data syncing is finished'
+ '[' 'Data syncing is finished' '!=' 'Data syncing is finished' ']'
+ cleanup_process cdc.test
wait process cdc.test exit for 1-th time...
wait process cdc.test exit for 2-th time...
wait process cdc.test exit for 3-th time...
cdc.test: no process found
wait process cdc.test exit for 4-th time...
process cdc.test already exit
+ stop_tidb_cluster
+ run_case_with_failpoint conf/changefeed-redo.toml
+ rm -rf /tmp/tidb_cdc_test/synced_status_with_redo
+ mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo
+ start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
The 1 times to try to start tidb cluster...
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 723 100 723 0 0 11781 0 --:--:-- --:--:-- --:--:-- 11852
+ synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-07 15:32:07.450","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-05-07 15:32:34.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}'
++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-07' '15:32:07.450","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-07' '15:32:34.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}'
++ jq .synced
+ status=false
+ '[' false '!=' false ']'
++ jq -r .info
++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-07' '15:32:07.450","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-07' '15:32:34.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}'
+ info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait'
+ target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait'
+ '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']'
+ export GO_FAILPOINTS=
+ GO_FAILPOINTS=
+ cleanup_process cdc.test
wait process cdc.test exit for 1-th time...
wait process cdc.test exit for 2-th time...
wait process cdc.test exit for 3-th time...
cdc.test: no process found
wait process cdc.test exit for 4-th time...
process cdc.test already exit
+ stop_tidb_cluster
chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo
Starting Upstream PD...
Release Version: v8.1.0-alpha-20-g13eaf8aaa
Edition: Community
Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d
Git Branch: release-8.1
UTC Build Time: 2024-04-30 07:22:55
Starting Downstream PD...
Release Version: v8.1.0-alpha-20-g13eaf8aaa
Edition: Community
Git Commit Hash: 13eaf8aaa661dc0d3885ac9e59145f69752cee2d
Git Branch: release-8.1
UTC Build Time: 2024-04-30 07:22:55
Verifying upstream PD is started...
Verifying downstream PD is started...
Starting Upstream TiKV...
TiKV
Release Version: 8.1.0
Edition: Community
Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8
Git Commit Branch: release-8.1
UTC Build Time: 2024-04-30 06:15:40
Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27)
Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored
Profile: dist_release
Starting Downstream TiKV...
TiKV
Release Version: 8.1.0
Edition: Community
Git Commit Hash: 56613f7c3e28c02853cc51d15bc1b77f68b58be8
Git Commit Branch: release-8.1
UTC Build Time: 2024-04-30 06:15:40
Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27)
Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored
Profile: dist_release
Starting Upstream TiDB...
Release Version: v8.1.0-alpha-106-gc8e6662d10
Edition: Community
Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991
Git Branch: release-8.1
UTC Build Time: 2024-05-06 10:39:26
GoVersion: go1.21.6
Race Enabled: false
Check Table Before Drop: false
Store: unistore
Starting Downstream TiDB...
Release Version: v8.1.0-alpha-106-gc8e6662d10
Edition: Community
Git Commit Hash: c8e6662d10deed030ef864169ae27718f5a61991
Git Branch: release-8.1
UTC Build Time: 2024-05-06 10:39:26
GoVersion: go1.21.6
Race Enabled: false
Check Table Before Drop: false
Store: unistore
Verifying Upstream TiDB is started...
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
+ check_logs /tmp/tidb_cdc_test/synced_status
++ date
+ echo '[Tue May 7 15:32:46 CST 2024] <<<<<< run test case synced_status success! >>>>>>'
[Tue May 7 15:32:46 CST 2024] <<<<<< run test case synced_status success! >>>>>>
+ stop_tidb_cluster
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
VARIABLE_NAME VARIABLE_VALUE COMMENT
bootstrapped True Bootstrap flag. Do not delete.
tidb_server_version 196 Bootstrap version. Do not delete.
system_tz Asia/Shanghai TiDB Global System Timezone.
new_collation_enabled True If the new collations are enabled. Do not edit it.
ddl_table_version 3 DDL Table Version. Do not delete.
tikv_gc_leader_uuid 63d47e07758000f Current GC worker leader UUID. (DO NOT EDIT)
tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-77-16j3h, pid:19597, start at 2024-05-07 15:32:46.18026547 +0800 CST m=+5.270045764 Host name and pid of current GC leader. (DO NOT EDIT)
tikv_gc_leader_lease 20240507-15:34:46.187 +0800 Current GC worker leader lease. (DO NOT EDIT)
tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used
tikv_gc_enable true Current GC enable status
tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format.
tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format.
tikv_gc_last_run_time 20240507-15:32:46.166 +0800 The time when last GC starts. (DO NOT EDIT)
tikv_gc_safe_point 20240507-15:22:46.166 +0800 All versions after safe point can be accessed. (DO NOT EDIT)
VARIABLE_NAME VARIABLE_VALUE COMMENT
bootstrapped True Bootstrap flag. Do not delete.
tidb_server_version 196 Bootstrap version. Do not delete.
system_tz Asia/Shanghai TiDB Global System Timezone.
new_collation_enabled True If the new collations are enabled. Do not edit it.
ddl_table_version 3 DDL Table Version. Do not delete.
tikv_gc_leader_uuid 63d47e07758000f Current GC worker leader UUID. (DO NOT EDIT)
tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-77-16j3h, pid:19597, start at 2024-05-07 15:32:46.18026547 +0800 CST m=+5.270045764 Host name and pid of current GC leader. (DO NOT EDIT)
tikv_gc_leader_lease 20240507-15:34:46.187 +0800 Current GC worker leader lease. (DO NOT EDIT)
tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used
tikv_gc_enable true Current GC enable status
tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format.
tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format.
tikv_gc_last_run_time 20240507-15:32:46.166 +0800 The time when last GC starts. (DO NOT EDIT)
tikv_gc_safe_point 20240507-15:22:46.166 +0800 All versions after safe point can be accessed. (DO NOT EDIT)
Verifying Downstream TiDB is started...
VARIABLE_NAME VARIABLE_VALUE COMMENT
bootstrapped True Bootstrap flag. Do not delete.
tidb_server_version 196 Bootstrap version. Do not delete.
system_tz Asia/Shanghai TiDB Global System Timezone.
new_collation_enabled True If the new collations are enabled. Do not edit it.
ddl_table_version 3 DDL Table Version. Do not delete.
tikv_gc_leader_uuid 63d47e077400015 Current GC worker leader UUID. (DO NOT EDIT)
tikv_gc_leader_desc host:ap-tiflow-release-8-1-pull-cdc-integration-pulsar-test-77-16j3h, pid:19684, start at 2024-05-07 15:32:46.20247597 +0800 CST m=+5.241892193 Host name and pid of current GC leader. (DO NOT EDIT)
tikv_gc_leader_lease 20240507-15:34:46.209 +0800 Current GC worker leader lease. (DO NOT EDIT)
tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used
tikv_gc_enable true Current GC enable status
tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format.
tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format.
tikv_gc_last_run_time 20240507-15:32:46.211 +0800 The time when last GC starts. (DO NOT EDIT)
tikv_gc_safe_point 20240507-15:22:46.211 +0800 All versions after safe point can be accessed. (DO NOT EDIT)
Starting Upstream TiFlash...
TiFlash
Release Version: v8.1.0-alpha-20-geb96e7201
Edition: Community
Git Commit Hash: eb96e720183ea798458afc99cfb034864ed75c4c
Git Branch: HEAD
UTC Build Time: 2024-05-05 06:18:35
Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto
Profile: RELWITHDEBINFO
Compiler: clang++ 13.0.0
Raft Proxy
Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79
Git Commit Branch: HEAD
UTC Build Time: 2024-05-05 06:22:21
Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14)
Storage Engine: tiflash
Prometheus Prefix: tiflash_proxy_
Profile: release
Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored
Verifying Upstream TiFlash is started...
Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log
Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log
arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v8.1.0-alpha-20-geb96e7201"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["eb96e720183ea798458afc99cfb034864ed75c4c"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
\033[0;36m<<< Run all test success >>>\033[0m
[Pipeline] }
Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-77/tiflow-cdc already exists)
[Pipeline] // cache
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
+ cd /tmp/tidb_cdc_test/synced_status_with_redo
+ export 'GO_FAILPOINTS=github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)'
+ GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)'
++ run_cdc_cli_tso_query 127.0.0.1 2379
+ pd_host=127.0.0.1
+ pd_port=2379
+ is_tls=false
+ '[' false == true ']'
++ run_cdc_cli tso query --pd=http://127.0.0.1:2379
+ cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.20966.out cli tso query --pd=http://127.0.0.1:2379
+ set +x
+ tso='449594568623259650
PASS
coverage: 2.0% of statements in github.com/pingcap/tiflow/...'
+ echo 449594568623259650 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/...
+ awk -F ' ' '{print $1}'
+ set +x
+ start_ts=449594568623259650
+ run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test
[Tue May 7 15:32:53 CST 2024] <<<<<< START cdc server in synced_status_with_redo case >>>>>>
+ [[ '' == \t\r\u\e ]]
+ set +e
+ get_info_fail_msg='failed to get info:'
+ etcd_info_msg='etcd info'
+ '[' -z '' ']'
+ curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL'
+ [[ no != \n\o ]]
+ (( i = 0 ))
+ (( i <= 50 ))
+ GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)'
+ cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.2100221004.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default
++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL
* About to connect() to 127.0.0.1 port 8300 (#0)
* Trying 127.0.0.1...
* Connection refused
* Failed connect to 127.0.0.1:8300; Connection refused
* Closing connection 0
+ res=
+ echo ''
+ grep -q 'failed to get info:'
+ echo ''
+ grep -q 'etcd info'
+ '[' 0 -eq 50 ']'
+ sleep 3
+ (( i++ ))
+ (( i <= 50 ))
++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL
* About to connect() to 127.0.0.1 port 8300 (#0)
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0)
* Server auth using Basic with user 'ticdc'
> GET /debug/info HTTP/1.1
> Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:8300
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Tue, 07 May 2024 07:32:56 GMT
< Content-Length: 828
< Content-Type: text/plain; charset=utf-8
<
{ [data not shown]
* Connection #0 to host 127.0.0.1 left intact
+ res='
*** owner info ***:
*** processors info ***:
*** etcd info ***:
/tidb/cdc/default/__cdc_meta__/capture/0c86674c-c79c-432e-818a-01577f2b683d
{"id":"0c86674c-c79c-432e-818a-01577f2b683d","address":"127.0.0.1:8300","version":"v8.1.0-alpha-26-g299703e05","git-hash":"299703e05932fae47116ddc8fcf22bdcb4e98ea7","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1715067173}
/tidb/cdc/default/__cdc_meta__/meta/meta-version
1
/tidb/cdc/default/__cdc_meta__/owner/22318f51f7fb3f3a
0c86674c-c79c-432e-818a-01577f2b683d
/tidb/cdc/default/default/upstream/7366157350011748822
{"id":7366157350011748822,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}'
+ echo '
*** owner info ***:
*** processors info ***:
*** etcd info ***:
/tidb/cdc/default/__cdc_meta__/capture/0c86674c-c79c-432e-818a-01577f2b683d
{"id":"0c86674c-c79c-432e-818a-01577f2b683d","address":"127.0.0.1:8300","version":"v8.1.0-alpha-26-g299703e05","git-hash":"299703e05932fae47116ddc8fcf22bdcb4e98ea7","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1715067173}
/tidb/cdc/default/__cdc_meta__/meta/meta-version
1
/tidb/cdc/default/__cdc_meta__/owner/22318f51f7fb3f3a
0c86674c-c79c-432e-818a-01577f2b683d
/tidb/cdc/default/default/upstream/7366157350011748822
{"id":7366157350011748822,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}'
+ grep -q 'failed to get info:'
+ echo '
*** owner info ***:
*** processors info ***:
*** etcd info ***:
/tidb/cdc/default/__cdc_meta__/capture/0c86674c-c79c-432e-818a-01577f2b683d
{"id":"0c86674c-c79c-432e-818a-01577f2b683d","address":"127.0.0.1:8300","version":"v8.1.0-alpha-26-g299703e05","git-hash":"299703e05932fae47116ddc8fcf22bdcb4e98ea7","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/bin/cdc.test","start-timestamp":1715067173}
/tidb/cdc/default/__cdc_meta__/meta/meta-version
1
/tidb/cdc/default/__cdc_meta__/owner/22318f51f7fb3f3a
0c86674c-c79c-432e-818a-01577f2b683d
/tidb/cdc/default/default/upstream/7366157350011748822
{"id":7366157350011748822,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}'
+ grep -q 'etcd info'
+ break
+ set +x
+ config_path=conf/changefeed-redo.toml
+ SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1'
+ run_cdc_cli changefeed create --start-ts=449594568623259650 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml
+ cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.21065.out cli changefeed create --start-ts=449594568623259650 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-8.1/pull_cdc_integration_pulsar_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml
Create changefeed successfully!
ID: test-1
Info: {"upstream_id":7366157350011748822,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-05-07T15:32:56.611567622+08:00","start_ts":449594568623259650,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v8.1.0-alpha-26-g299703e05","resolved_ts":449594568623259650,"checkpoint_ts":449594568623259650,"checkpoint_time":"2024-05-07 15:32:51.567"}
PASS
coverage: 2.7% of statements in github.com/pingcap/tiflow/...
+ set +x
+ sleep 20
++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 723 100 723 0 0 4671 0 --:--:-- --:--:-- --:--:-- 4694
+ synced_status='{"synced":false,"sink_checkpoint_ts":"2024-05-07 15:32:51.567","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-05-07 15:33:18.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}'
++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-07' '15:32:51.567","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-07' '15:33:18.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}'
++ jq .synced
+ status=false
+ '[' false '!=' false ']'
++ echo '{"synced":false,"sink_checkpoint_ts":"2024-05-07' '15:32:51.567","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-05-07' '15:33:18.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}'
++ jq -r .info
+ info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait'
+ target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait'
+ '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']'
+ export GO_FAILPOINTS=
+ GO_FAILPOINTS=
+ cleanup_process cdc.test
wait process cdc.test exit for 1-th time...
wait process cdc.test exit for 2-th time...
wait process cdc.test exit for 3-th time...
cdc.test: no process found
wait process cdc.test exit for 4-th time...
process cdc.test already exit
+ stop_tidb_cluster
+ check_logs /tmp/tidb_cdc_test/synced_status_with_redo
++ date
+ echo '[Tue May 7 15:33:29 CST 2024] <<<<<< run test case synced_status_with_redo success! >>>>>>'
[Tue May 7 15:33:29 CST 2024] <<<<<< run test case synced_status_with_redo success! >>>>>>
+ stop_tidb_cluster
\033[0;36m<<< Run all test success >>>\033[0m
[Pipeline] }
Cache not saved (ws/jenkins-pingcap-tiflow-release-8.1-pull_cdc_integration_pulsar_test-77/tiflow-cdc already exists)
[Pipeline] // cache
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Finished: SUCCESS
![[Jenkins]](/jenkins/static/62b7ca6c/images/svgs/logo.svg){kind=logo}
