Skip to content
[Jenkins]Jenkins
log in

Console Output

Skipping 1,055 KB.. Full Log
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*===== TEST: [lightning_restore] success! =====*
{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-86-gb421b729c8-dirty","git_hash":"b421b729c8a7c43ad7250f38e655cdda03d0ca77"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }, "log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
Run cases: lightning_routes
TiFlash started.
selected test cases: lightning_duplicate_resolution_error
Starting PD...
*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error/run.sh... =====*
+ check_cluster_version 5 2 0 'duplicate detection'
++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error/run.sh
+ mydir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error
+ run_sql 'DROP TABLE IF EXISTS dup_resolve.a'
+ run_sql 'DROP TABLE IF EXISTS lightning_task_info.conflict_error_v3'
+ run_lightning --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error/config.toml
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:40:02 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error --backend' local --config '/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error/config.toml >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_duplicate_resolution_error.89202.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error/data --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_resolution_error.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error/config.toml
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

tidb lightning encountered error: [Lightning:Restore:ErrFoundDataConflictRecords]found data conflict records in table a, primary key is '3', row data is '(3, 3, "3.csv")'
warning: GOCOVERDIR not set, no coverage data emitted
+ '[' 0 -eq 0 ']'
+ tail -n 10 /tmp/lightning_test/lightning.log
+ grep ERROR
+ tail -n 1
+ grep -Fq '[Lightning:Restore:ErrFoundDataConflictRecords]found data conflict records in table a, primary key is '\''3'\'', row data is '\''(3, 3, \"3.csv\")'\'''
+ check_not_contains 'the whole procedure completed' /tmp/lightning_test/lightning.log
*===== TEST: [lightning_duplicate_resolution_error] success! =====*
tidb lightning exit successfully
PASS
coverage: 27.6% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ run_sql 'SELECT count(*), sum(c) FROM cpeng.a'
*************************** 1. row ***************************
count(*): 4
  sum(c): 10
+ check_contains 'count(*): 4'
+ check_contains 'sum(c): 10'
+ run_sql 'SELECT count(*), sum(c) FROM cpeng.b'
*************************** 1. row ***************************
count(*): 4
  sum(c): 46
+ check_contains 'count(*): 4'
+ check_contains 'sum(c): 46'
+ run_sql 'DROP DATABASE cpeng;'
+ rm -f /tmp/tidb_lightning_checkpoint_local_backend_test.pb
+ set +e
+ export 'GO_FAILPOINTS=github.com/pingcap/tidb/lightning/pkg/importer/FailIfStatusBecomes=return(90);'
+ GO_FAILPOINTS='github.com/pingcap/tidb/lightning/pkg/importer/FailIfStatusBecomes=return(90);'
++ seq 6
+ for i in '$(seq "$ENGINE_COUNT")'
+ echo '******** Importing Table Now (step 1/6) ********'
******** Importing Table Now (step 1/6) ********
+ run_lightning --backend local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:40:03 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend --backend' local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config '/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_local_backend.27664.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/data --sorted-kv-dir /tmp/lightning_test/lightning_local_backend.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml
bad failpoint ""
+ '[' 1 -ne 0 ']'
+ for i in '$(seq "$ENGINE_COUNT")'
+ echo '******** Importing Table Now (step 2/6) ********'
******** Importing Table Now (step 2/6) ********
+ run_lightning --backend local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:40:03 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend --backend' local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config '/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_local_backend.27686.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/data --sorted-kv-dir /tmp/lightning_test/lightning_local_backend.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml
bad failpoint ""
+ '[' 1 -ne 0 ']'
+ for i in '$(seq "$ENGINE_COUNT")'
+ echo '******** Importing Table Now (step 3/6) ********'
******** Importing Table Now (step 3/6) ********
+ run_lightning --backend local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:40:03 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend --backend' local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config '/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_local_backend.27709.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/data --sorted-kv-dir /tmp/lightning_test/lightning_local_backend.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml
bad failpoint ""
+ '[' 1 -ne 0 ']'
+ for i in '$(seq "$ENGINE_COUNT")'
+ echo '******** Importing Table Now (step 4/6) ********'
******** Importing Table Now (step 4/6) ********
+ run_lightning --backend local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:40:03 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend --backend' local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config '/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_local_backend.27732.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/data --sorted-kv-dir /tmp/lightning_test/lightning_local_backend.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml
bad failpoint ""
+ '[' 1 -ne 0 ']'
+ for i in '$(seq "$ENGINE_COUNT")'
+ echo '******** Importing Table Now (step 5/6) ********'
******** Importing Table Now (step 5/6) ********
+ run_lightning --backend local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:40:03 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend --backend' local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config '/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_local_backend.27756.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/data --sorted-kv-dir /tmp/lightning_test/lightning_local_backend.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml
bad failpoint ""
+ '[' 1 -ne 0 ']'
+ for i in '$(seq "$ENGINE_COUNT")'
+ echo '******** Importing Table Now (step 6/6) ********'
******** Importing Table Now (step 6/6) ********
+ run_lightning --backend local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:40:03 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend --backend' local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config '/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_local_backend.27780.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/data --sorted-kv-dir /tmp/lightning_test/lightning_local_backend.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml
bad failpoint ""
+ '[' 1 -ne 0 ']'
+ set -e
+ export GO_FAILPOINTS=
+ GO_FAILPOINTS=
+ echo '******** Verify checkpoint no-op ********'
******** Verify checkpoint no-op ********
+ run_lightning --backend local --enable-checkpoint=1 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml --log-file /tmp/lightning_test/sql_res.lightning_local_backend.txt -L debug
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:40:03 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend --backend' local --enable-checkpoint=1 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml --log-file /tmp/lightning_test/sql_res.lightning_local_backend.txt -L 'debug >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_local_backend.27804.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/data --sorted-kv-dir /tmp/lightning_test/lightning_local_backend.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --enable-checkpoint=1 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml --log-file /tmp/lightning_test/sql_res.lightning_local_backend.txt -L debug
Verbose debug logs will be written to /tmp/lightning_test/sql_res.lightning_local_backend.txt

{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-86-gb421b729c8-dirty","git_hash":"b421b729c8a7c43ad7250f38e655cdda03d0ca77"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
tidb lightning exit successfully
PASS
coverage: 27.1% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ check_not_contains 'failed to set system var'
+ check_not_contains 'unknown system var'
+ check_contains 'skip read-only variable'
+ check_contains lc_time_names
+ run_sql 'SELECT count(*), sum(c) FROM cpeng.a'
*************************** 1. row ***************************
count(*): 4
  sum(c): 10
+ check_contains 'count(*): 4'
+ check_contains 'sum(c): 10'
+ run_sql 'SELECT count(*), sum(c) FROM cpeng.b'
*************************** 1. row ***************************
count(*): 4
  sum(c): 46
+ check_contains 'count(*): 4'
+ check_contains 'sum(c): 46'
+ for ckpt in mysql file
+ run_sql 'DROP DATABASE IF EXISTS cpeng;'
+ run_sql 'DROP DATABASE IF EXISTS tidb_lightning_checkpoint_local_backend_test'
+ rm -f /tmp/tidb_lightning_checkpoint_local_backend_test.pb
+ set +e
+ export GO_FAILPOINTS=github.com/pingcap/tidb/lightning/pkg/importer/FailAfterWriteRows=return
+ GO_FAILPOINTS=github.com/pingcap/tidb/lightning/pkg/importer/FailAfterWriteRows=return
+ run_lightning --backend local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/mysql.toml
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:40:05 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend --backend' local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config '/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/mysql.toml >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_local_backend.27901.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/data --sorted-kv-dir /tmp/lightning_test/lightning_local_backend.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/mysql.toml
Verbose debug logs will be written to /tmp/lightning_test/lightning-local.log

Run cases: lightning_duplicate_resolution_error_pk_multiple_files
tidb lightning exit successfully
PASS
coverage: 24.5% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ set -e
+ run_lightning_ctl --check-local-storage --backend local --enable-checkpoint=1 --config=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/mysql.toml
+ grep -Fq 'No table has lost intermediate files according to given config' /tmp/lightning_test/lightning_ctl.output
+ run_sql 'DROP DATABASE IF EXISTS cpeng;'
+ run_sql 'DROP DATABASE IF EXISTS tidb_lightning_checkpoint_local_backend_test'
+ rm -f /tmp/tidb_lightning_checkpoint_local_backend_test.pb
+ set +e
+ export 'GO_FAILPOINTS=github.com/pingcap/tidb/lightning/pkg/importer/LocalBackendSaveCheckpoint=return;github.com/pingcap/tidb/lightning/pkg/importer/FailIfImportedChunk=return'
+ GO_FAILPOINTS='github.com/pingcap/tidb/lightning/pkg/importer/LocalBackendSaveCheckpoint=return;github.com/pingcap/tidb/lightning/pkg/importer/FailIfImportedChunk=return'
+ run_lightning --backend local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/mysql.toml
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:40:06 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend --backend' local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config '/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/mysql.toml >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_local_backend.28010.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/data --sorted-kv-dir /tmp/lightning_test/lightning_local_backend.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/mysql.toml
Verbose debug logs will be written to /tmp/lightning_test/lightning-local.log

TiFlash started.
*===== TEST: [lightning_pd_leader_switch] success! =====*
panic: forcing failure due to FailIfImportedChunk

goroutine 373 [running]:
github.com/pingcap/tidb/lightning/pkg/importer.(*Controller).listenCheckpointUpdates(0xc000e8da00, {0xc000ea9950?})
	/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/pkg/importer/import.go:898 +0xb66
created by github.com/pingcap/tidb/lightning/pkg/importer.(*Controller).initCheckpoint in goroutine 131
	/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/pkg/importer/import.go:642 +0x2ad
+ set -e
+ run_lightning_ctl --check-local-storage --backend local --enable-checkpoint=1 --config=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/mysql.toml
+ grep -Eq 'These tables are missing intermediate files: \[.+\]' /tmp/lightning_test/lightning_ctl.output
+ ls -lA /tmp/lightning_test/lightning_local_backend.sorted
total 16
drwxr-xr-x. 2 jenkins jenkins 4096 May  6 20:40 0f3182ea-38f9-5947-825c-2db765ea3f98
drwxr-x---. 2 jenkins jenkins 4096 May  6 20:40 0f3182ea-38f9-5947-825c-2db765ea3f98.sst
drwxr-xr-x. 2 jenkins jenkins 4096 May  6 20:40 c872a8e3-b992-5cbf-b35b-2a89bc464ee4
drwxr-x---. 2 jenkins jenkins 4096 May  6 20:40 c872a8e3-b992-5cbf-b35b-2a89bc464ee4.sst
+ set +e
+ export 'GO_FAILPOINTS=github.com/pingcap/tidb/lightning/pkg/importer/FailIfIndexEngineImported=return(1)'
+ GO_FAILPOINTS='github.com/pingcap/tidb/lightning/pkg/importer/FailIfIndexEngineImported=return(1)'
+ run_lightning --backend local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/mysql.toml
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:40:07 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend --backend' local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config '/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/mysql.toml >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_local_backend.28106.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/data --sorted-kv-dir /tmp/lightning_test/lightning_local_backend.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/mysql.toml
Verbose debug logs will be written to /tmp/lightning_test/lightning-local.log

Starting PD...
2024/05/06 20:40:07 [JOB 1] WAL file /tmp/lightning_test/lightning_local_backend.sorted/c872a8e3-b992-5cbf-b35b-2a89bc464ee4/000002.log with log number 000002 stopped reading at offset: 0; replayed 0 keys in 0 batches
2024/05/06 20:40:07 [JOB 1] WAL file /tmp/lightning_test/lightning_local_backend.sorted/0f3182ea-38f9-5947-825c-2db765ea3f98/000002.log with log number 000002 stopped reading at offset: 0; replayed 0 keys in 0 batches
panic: forcing failure due to FailIfIndexEngineImported

goroutine 393 [running]:
github.com/pingcap/tidb/lightning/pkg/importer.(*Controller).listenCheckpointUpdates(0xc0006bbc00, {0xc001306eb0?})
	/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/pkg/importer/import.go:918 +0xab8
created by github.com/pingcap/tidb/lightning/pkg/importer.(*Controller).initCheckpoint in goroutine 146
	/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/pkg/importer/import.go:642 +0x2ad
+ set -e
+ run_lightning_ctl --check-local-storage --backend local --enable-checkpoint=1 --config=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/mysql.toml
+ grep -Fq 'No table has lost intermediate files according to given config' /tmp/lightning_test/lightning_ctl.output
+ for ckpt in mysql file
+ run_sql 'DROP DATABASE IF EXISTS cpeng;'
+ run_sql 'DROP DATABASE IF EXISTS tidb_lightning_checkpoint_local_backend_test'
+ rm -f /tmp/tidb_lightning_checkpoint_local_backend_test.pb
+ set +e
+ export GO_FAILPOINTS=github.com/pingcap/tidb/lightning/pkg/importer/FailAfterWriteRows=return
+ GO_FAILPOINTS=github.com/pingcap/tidb/lightning/pkg/importer/FailAfterWriteRows=return
+ run_lightning --backend local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/file.toml
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:40:08 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend --backend' local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config '/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/file.toml >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_local_backend.28205.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/data --sorted-kv-dir /tmp/lightning_test/lightning_local_backend.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/file.toml
Verbose debug logs will be written to /tmp/lightning_test/lightning-local.log

tidb lightning exit successfully
PASS
{
  "version": "v8.2.0-alpha-15-gf83febabe",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
coverage: 26.4% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ set -e
+ run_lightning_ctl --check-local-storage --backend local --enable-checkpoint=1 --config=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/file.toml
+ grep -Fq 'No table has lost intermediate files according to given config' /tmp/lightning_test/lightning_ctl.output
+ run_sql 'DROP DATABASE IF EXISTS cpeng;'
+ run_sql 'DROP DATABASE IF EXISTS tidb_lightning_checkpoint_local_backend_test'
+ rm -f /tmp/tidb_lightning_checkpoint_local_backend_test.pb
+ set +e
+ export 'GO_FAILPOINTS=github.com/pingcap/tidb/lightning/pkg/importer/LocalBackendSaveCheckpoint=return;github.com/pingcap/tidb/lightning/pkg/importer/FailIfImportedChunk=return'
+ GO_FAILPOINTS='github.com/pingcap/tidb/lightning/pkg/importer/LocalBackendSaveCheckpoint=return;github.com/pingcap/tidb/lightning/pkg/importer/FailIfImportedChunk=return'
+ run_lightning --backend local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/file.toml
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:40:09 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend --backend' local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config '/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/file.toml >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_local_backend.28314.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/data --sorted-kv-dir /tmp/lightning_test/lightning_local_backend.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/file.toml
Verbose debug logs will be written to /tmp/lightning_test/lightning-local.log

panic: forcing failure due to FailIfImportedChunk

goroutine 329 [running]:
github.com/pingcap/tidb/lightning/pkg/importer.(*Controller).listenCheckpointUpdates(0xc0008cf000, {0xc001800d70?})
	/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/pkg/importer/import.go:898 +0xb66
created by github.com/pingcap/tidb/lightning/pkg/importer.(*Controller).initCheckpoint in goroutine 130
	/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/pkg/importer/import.go:642 +0x2ad
+ set -e
+ run_lightning_ctl --check-local-storage --backend local --enable-checkpoint=1 --config=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/file.toml
+ grep -Eq 'These tables are missing intermediate files: \[.+\]' /tmp/lightning_test/lightning_ctl.output
+ ls -lA /tmp/lightning_test/lightning_local_backend.sorted
total 16
drwxr-xr-x. 2 jenkins jenkins 4096 May  6 20:40 0f3182ea-38f9-5947-825c-2db765ea3f98
drwxr-x---. 2 jenkins jenkins 4096 May  6 20:40 0f3182ea-38f9-5947-825c-2db765ea3f98.sst
drwxr-xr-x. 2 jenkins jenkins 4096 May  6 20:40 c872a8e3-b992-5cbf-b35b-2a89bc464ee4
drwxr-x---. 2 jenkins jenkins 4096 May  6 20:40 c872a8e3-b992-5cbf-b35b-2a89bc464ee4.sst
+ set +e
+ export 'GO_FAILPOINTS=github.com/pingcap/tidb/lightning/pkg/importer/FailIfIndexEngineImported=return(1)'
+ GO_FAILPOINTS='github.com/pingcap/tidb/lightning/pkg/importer/FailIfIndexEngineImported=return(1)'
+ run_lightning --backend local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/file.toml
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:40:09 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend --backend' local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config '/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/file.toml >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_local_backend.28414.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/data --sorted-kv-dir /tmp/lightning_test/lightning_local_backend.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-local.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/file.toml
Verbose debug logs will be written to /tmp/lightning_test/lightning-local.log

2024/05/06 20:40:10 [JOB 1] WAL file /tmp/lightning_test/lightning_local_backend.sorted/c872a8e3-b992-5cbf-b35b-2a89bc464ee4/000002.log with log number 000002 stopped reading at offset: 0; replayed 0 keys in 0 batches
2024/05/06 20:40:10 [JOB 1] WAL file /tmp/lightning_test/lightning_local_backend.sorted/0f3182ea-38f9-5947-825c-2db765ea3f98/000002.log with log number 000002 stopped reading at offset: 0; replayed 0 keys in 0 batches
panic: forcing failure due to FailIfIndexEngineImported

goroutine 340 [running]:
github.com/pingcap/tidb/lightning/pkg/importer.(*Controller).listenCheckpointUpdates(0xc000d50800, {0xc000b7e0c0?})
	/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/pkg/importer/import.go:918 +0xab8
created by github.com/pingcap/tidb/lightning/pkg/importer.(*Controller).initCheckpoint in goroutine 146
	/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/pkg/importer/import.go:642 +0x2ad
+ set -e
+ run_lightning_ctl --check-local-storage --backend local --enable-checkpoint=1 --config=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_local_backend/file.toml
+ grep -Fq 'No table has lost intermediate files according to given config' /tmp/lightning_test/lightning_ctl.output
+ rm -r /tmp/lightning_test/lightning_local_backend.sorted
*===== TEST: [lightning_local_backend] success! =====*
Run cases: lightning_add_index
Run cases: lightning_max_incr
Starting PD...
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
{
  "version": "v8.2.0-alpha-15-gf83febabe",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
Starting PD...
{
  "version": "v8.2.0-alpha-15-gf83febabe",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
{
  "version": "v8.2.0-alpha-15-gf83febabe",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-86-gb421b729c8-dirty","git_hash":"b421b729c8a7c43ad7250f38e655cdda03d0ca77"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }, "engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }, "log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
TiFlash started.
selected test cases: lightning_routes
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-86-gb421b729c8-dirty","git_hash":"b421b729c8a7c43ad7250f38e655cdda03d0ca77"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }, "engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_routes/run.sh... =====*
+ echo 'testing checksum-error...'
testing checksum-error...
+ export 'GO_FAILPOINTS=github.com/pingcap/tidb/lightning/pkg/importer/checksum-error=1*return()'
+ GO_FAILPOINTS='github.com/pingcap/tidb/lightning/pkg/importer/checksum-error=1*return()'
+ run_sql 'DROP DATABASE IF EXISTS routes_a0;'
+ run_sql 'DROP DATABASE IF EXISTS routes_a1;'
+ run_sql 'DROP DATABASE IF EXISTS routes_b;'
+ run_lightning
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:40:24 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_routes  >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_routes.26032.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_routes/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_routes/data --sorted-kv-dir /tmp/lightning_test/lightning_routes.sorted --enable-checkpoint=0 --check-requirements=0
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

tidb lightning exit successfully
PASS
coverage: 23.3% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ echo 'test checksum-error success!'
test checksum-error success!
+ run_sql 'SELECT count(1), sum(x) FROM routes_b.u;'
*************************** 1. row ***************************
count(1): 4
  sum(x): 259
+ check_contains 'count(1): 4'
+ check_contains 'sum(x): 259'
+ run_sql 'SELECT count(1), sum(x) FROM routes_a1.s1;'
*************************** 1. row ***************************
count(1): 1
  sum(x): 1296
+ check_contains 'count(1): 1'
+ check_contains 'sum(x): 1296'
+ run_sql 'SHOW TABLES IN routes_a1;'
*************************** 1. row ***************************
Tables_in_routes_a1: s1
+ check_not_contains 'Tables_in_routes_a1: t2'
*===== TEST: [lightning_routes] success! =====*
{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-86-gb421b729c8-dirty","git_hash":"b421b729c8a7c43ad7250f38e655cdda03d0ca77"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
TiFlash started.
selected test cases: lightning_duplicate_resolution_error_pk_multiple_files
Run cases: lightning_routes_panic
{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-86-gb421b729c8-dirty","git_hash":"b421b729c8a7c43ad7250f38e655cdda03d0ca77"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }, "log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
TiFlash started.
selected test cases: lightning_add_index
*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_pk_multiple_files/run.sh... =====*
+ check_cluster_version 5 2 0 'duplicate detection'
++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_pk_multiple_files/run.sh
+ mydir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_pk_multiple_files
+ run_sql 'DROP TABLE IF EXISTS dup_resolve.a'
+ run_sql 'DROP TABLE IF EXISTS lightning_task_info.conflict_error_v3'
+ run_lightning --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_pk_multiple_files/config.toml
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:40:29 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_pk_multiple_files --backend' local --config '/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_pk_multiple_files/config.toml >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_duplicate_resolution_error_pk_multiple_files.91071.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_pk_multiple_files/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_pk_multiple_files/data --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_resolution_error_pk_multiple_files.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_pk_multiple_files/config.toml
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

Starting PD...
tidb lightning encountered error: failed to record conflict errors: [Lightning:Restore:ErrFoundDataConflictRecords]found data conflict records in table a, primary key is '1', row data is '(1, 10, "10.csv")'
warning: GOCOVERDIR not set, no coverage data emitted
+ '[' 0 -eq 0 ']'
+ tail -n 10 /tmp/lightning_test/lightning.log
+ grep ERROR
+ tail -n 1
+ grep -Fq '[Lightning:Restore:ErrFoundDataConflictRecords]found data conflict records in table a'
+ check_not_contains 'the whole procedure completed' /tmp/lightning_test/lightning.log
*===== TEST: [lightning_duplicate_resolution_error_pk_multiple_files] success! =====*
*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/run.sh... =====*
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:40:31 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index --config' /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/config1.toml --log-file '/tmp/lightning_test/lightning-add-index1.log >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_add_index.23290.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/data --sorted-kv-dir /tmp/lightning_test/lightning_add_index.sorted --enable-checkpoint=0 --check-requirements=0 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/config1.toml --log-file /tmp/lightning_test/lightning-add-index1.log
Verbose debug logs will be written to /tmp/lightning_test/lightning-add-index1.log

TiFlash started.
selected test cases: lightning_max_incr
tidb lightning exit successfully
PASS
coverage: 23.2% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
Run cases: lightning_duplicate_resolution_error_uk_multiple_files
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:40:32 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index --config' /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/config2.toml --log-file '/tmp/lightning_test/lightning-add-index2.log >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_add_index.23381.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/data --sorted-kv-dir /tmp/lightning_test/lightning_add_index.sorted --enable-checkpoint=0 --check-requirements=0 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/config2.toml --log-file /tmp/lightning_test/lightning-add-index2.log
Verbose debug logs will be written to /tmp/lightning_test/lightning-add-index2.log

{
  "version": "v8.2.0-alpha-15-gf83febabe",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_max_incr/run.sh... =====*
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_max_incr/run.sh
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_max_incr
++ pwd
+ CUR=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_max_incr
+ check_cluster_version 4 0 0 'local backend'
+ ENGINE_COUNT=6
+ cleanup
+ rm -f /tmp/lightning_test/lightning.log
+ rm -rf /tmp/lightning_test/sst
+ run_sql 'DROP DATABASE IF EXISTS db;'
+ run_lightning --sorted-kv-dir /tmp/lightning_test/sst --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_max_incr/config.toml --log-file /tmp/lightning_test/lightning.log
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:40:33 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_max_incr --sorted-kv-dir' /tmp/lightning_test/sst --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_max_incr/config.toml --log-file '/tmp/lightning_test/lightning.log >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_max_incr.30294.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_max_incr/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_max_incr/data --sorted-kv-dir /tmp/lightning_test/lightning_max_incr.sorted --enable-checkpoint=0 --check-requirements=0 --sorted-kv-dir /tmp/lightning_test/sst --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_max_incr/config.toml --log-file /tmp/lightning_test/lightning.log
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

tidb lightning exit successfully
PASS
coverage: 23.2% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ check_result
+ run_sql 'SHOW DATABASES;'
*************************** 1. row ***************************
Database: INFORMATION_SCHEMA
*************************** 2. row ***************************
Database: METRICS_SCHEMA
*************************** 3. row ***************************
Database: PERFORMANCE_SCHEMA
*************************** 4. row ***************************
Database: db
*************************** 5. row ***************************
Database: mysql
*************************** 6. row ***************************
Database: sys
*************************** 7. row ***************************
Database: test
+ check_contains 'Database: db'
+ run_sql 'SHOW TABLES IN db;'
*************************** 1. row ***************************
Tables_in_db: test
*************************** 2. row ***************************
Tables_in_db: test1
+ check_contains 'Tables_in_db: test'
+ check_contains 'Tables_in_db: test1'
+ run_sql 'SELECT count(*) FROM db.test;'
*************************** 1. row ***************************
count(*): 2
+ check_contains 'count(*): 2'
+ run_sql 'SELECT count(*) FROM db.test1;'
*************************** 1. row ***************************
count(*): 2
+ check_contains 'count(*): 2'
+ run_sql 'INSERT INTO db.test(b) VALUES(11);'
+ run_sql 'INSERT INTO db.test1(b) VALUES(22);'
+ tee -a /tmp/lightning_test/sql_res.lightning_max_incr.txt
ERROR 1467 (HY000) at line 1: Failed to read auto-increment value from storage engine
+ check_contains ERROR
+ cleanup
+ rm -f /tmp/lightning_test/lightning.log
+ rm -rf /tmp/lightning_test/sst
+ run_sql 'DROP DATABASE IF EXISTS db;'
*===== TEST: [lightning_max_incr] success! =====*
Starting PD...
[Pipeline] }
Cache not saved (ws/jenkins-pingcap-tidb-merged_integration_lightning_test-538/lightning-tests already exists)
[Pipeline] // cache
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
{
  "version": "v8.2.0-alpha-15-gf83febabe",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-86-gb421b729c8-dirty","git_hash":"b421b729c8a7c43ad7250f38e655cdda03d0ca77"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }, "log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
TiFlash started.
selected test cases: lightning_routes_panic
*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_routes_panic/run.sh... =====*
+ run_sql 'DROP DATABASE IF EXISTS test1;'
+ run_sql 'DROP DATABASE IF EXISTS test;'
+ run_sql 'CREATE DATABASE test1;'
+ run_sql 'CREATE DATABASE test;'
+ run_sql 'CREATE TABLE test1.dump_test (x real primary key);'
+ run_sql 'CREATE TABLE test.u (x real primary key);'
+ run_lightning
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:40:50 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_routes_panic  >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_routes_panic.27941.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_routes_panic/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_routes_panic/data --sorted-kv-dir /tmp/lightning_test/lightning_routes_panic.sorted --enable-checkpoint=0 --check-requirements=0
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-86-gb421b729c8-dirty","git_hash":"b421b729c8a7c43ad7250f38e655cdda03d0ca77"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
tidb lightning exit successfully
PASS
coverage: 22.7% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ run_sql 'SELECT sum(x) FROM test.u;'
*************************** 1. row ***************************
sum(x): 43
+ check_contains 'sum(x): 43'
*===== TEST: [lightning_routes_panic] success! =====*
TiFlash started.
selected test cases: lightning_duplicate_resolution_error_uk_multiple_files
Run cases: lightning_row-format-v2
*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files/run.sh... =====*
+ check_cluster_version 5 2 0 'duplicate detection'
++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files/run.sh
+ mydir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files
+ run_sql 'DROP TABLE IF EXISTS dup_resolve.a'
+ run_sql 'DROP TABLE IF EXISTS lightning_task_info.conflict_error_v3'
+ run_lightning --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files/config.toml
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:40:54 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files --backend' local --config '/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files/config.toml >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_duplicate_resolution_error_uk_multiple_files.92938.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files/data --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_resolution_error_uk_multiple_files.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files/config.toml
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

Starting PD...
tidb lightning encountered error: [Lightning:Restore:ErrFoundIndexConflictRecords]found index conflict records in table a, index name is 'a.key_b', unique key is '[101]', primary key is '1'
warning: GOCOVERDIR not set, no coverage data emitted
+ '[' 0 -eq 0 ']'
+ tail -n 10 /tmp/lightning_test/lightning.log
+ grep ERROR
+ tail -n 1
+ grep -Fq '[Lightning:Restore:ErrFoundIndexConflictRecords]found index conflict records in table a, index name is '\''a.key_b'\'', unique key is '\''[101]'\'', primary key is '\''1'\'''
+ check_not_contains 'the whole procedure completed' /tmp/lightning_test/lightning.log
*===== TEST: [lightning_duplicate_resolution_error_uk_multiple_files] success! =====*
Run cases: lightning_duplicate_resolution_error_uk_multiple_files_multicol_index
{
  "version": "v8.2.0-alpha-15-gf83febabe",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
Starting PD...
{
  "version": "v8.2.0-alpha-15-gf83febabe",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-86-gb421b729c8-dirty","git_hash":"b421b729c8a7c43ad7250f38e655cdda03d0ca77"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }, "engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
tidb lightning exit successfully
PASS
coverage: 24.4% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
*************************** 1. row ***************************
       Table: multi_indexes
Create Table: CREATE TABLE `multi_indexes` (
  `c1` bigint(20) NOT NULL,
  `c2` varchar(255) NOT NULL,
  `c3` varchar(255) NOT NULL,
  `c4` varchar(255) NOT NULL,
  `c5` varchar(255) NOT NULL,
  `c6` varchar(255) NOT NULL,
  `c7` varchar(255) NOT NULL,
  `c8` varchar(255) NOT NULL,
  `c9` varchar(255) NOT NULL,
  `c10` varchar(255) NOT NULL,
  `c11` varchar(255) NOT NULL,
  PRIMARY KEY (`c1`) /*T![clustered_index] CLUSTERED */,
  KEY `idx_c2` (`c2`) COMMENT 'single column index',
  KEY `idx_c2_c3` (`c2`,`c3`) COMMENT 'multiple column index',
  UNIQUE KEY `uniq_c4` (`c4`) COMMENT 'single column unique key',
  UNIQUE KEY `uniq_c4_c5` (`c4`,`c5`) COMMENT 'multiple column unique key',
  KEY `idx_c6` (`c6`) COMMENT 'single column index with asc order',
  KEY `idx_c7` (`c7`) COMMENT 'single column index with desc order',
  KEY `idx_c6_c7` (`c6`,`c7`) COMMENT 'multiple column index with asc and desc order',
  KEY `idx_c8` (`c8`) COMMENT 'single column index with visible',
  KEY `idx_c9` (`c9`) /*!80000 INVISIBLE */ COMMENT 'single column index with invisible',
  KEY `idx_lower_c10` ((lower(`c10`))) COMMENT 'single column index with function',
  KEY `idx_prefix_c11` (`c11`(3)) COMMENT 'single column index with prefix',
  UNIQUE KEY `c2` (`c2`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:41:10 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index --enable-checkpoint=1' --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/config2.toml --log-file '/tmp/lightning_test/lightning-add-index2.log >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_add_index.23519.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/data --sorted-kv-dir /tmp/lightning_test/lightning_add_index.sorted --enable-checkpoint=0 --check-requirements=0 --enable-checkpoint=1 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/config2.toml --log-file /tmp/lightning_test/lightning-add-index2.log
Verbose debug logs will be written to /tmp/lightning_test/lightning-add-index2.log

TiFlash started.
selected test cases: lightning_row-format-v2
*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_row-format-v2/run.sh... =====*
+ run_sql 'show variables like "%tidb_row_format_version%";'
*************************** 1. row ***************************
Variable_name: tidb_row_format_version
        Value: 2
++ grep 'Value: [0-9]' /tmp/lightning_test/sql_res.lightning_row-format-v2.txt
++ awk '{print $2}'
+ row_format=2
+ '[' 2 -ne 2 ']'
+ run_sql 'DROP DATABASE IF EXISTS rowformatv2;'
+ run_lightning
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:41:14 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_row-format-v2  >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_row-format-v2.29803.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_row-format-v2/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_row-format-v2/data --sorted-kv-dir /tmp/lightning_test/lightning_row-format-v2.sorted --enable-checkpoint=0 --check-requirements=0
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-86-gb421b729c8-dirty","git_hash":"b421b729c8a7c43ad7250f38e655cdda03d0ca77"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }, "log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
tidb lightning canceled
PASS
coverage: 25.3% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:41:14 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index --enable-checkpoint=1' --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/config2.toml --log-file '/tmp/lightning_test/lightning-add-index2.log >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_add_index.23582.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/data --sorted-kv-dir /tmp/lightning_test/lightning_add_index.sorted --enable-checkpoint=0 --check-requirements=0 --enable-checkpoint=1 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/config2.toml --log-file /tmp/lightning_test/lightning-add-index2.log
Verbose debug logs will be written to /tmp/lightning_test/lightning-add-index2.log

tidb lightning exit successfully
PASS
coverage: 22.8% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ run_sql 'SELECT count(1) FROM rowformatv2.t1;'
*************************** 1. row ***************************
count(1): 50
+ check_contains 'count(1): 50'
+ run_sql 'SELECT DISTINCT col14 FROM rowformatv2.t1;'
*************************** 1. row ***************************
col14: NULL
*************************** 2. row ***************************
col14: 39
+ check_contains 'col14: NULL'
+ check_contains 'col14: 39'
+ '[' 2 -ne 2 ']'
*===== TEST: [lightning_row-format-v2] success! =====*
TiFlash started.
selected test cases: lightning_duplicate_resolution_error_uk_multiple_files_multicol_index
Run cases: lightning_s3
*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files_multicol_index/run.sh... =====*
+ check_cluster_version 5 2 0 'duplicate detection'
++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files_multicol_index/run.sh
+ mydir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files_multicol_index
+ run_sql 'DROP TABLE IF EXISTS dup_resolve.a'
+ run_sql 'DROP TABLE IF EXISTS lightning_task_info.conflict_error_v3'
+ run_lightning --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files_multicol_index/config.toml
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:41:18 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files_multicol_index --backend' local --config '/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files_multicol_index/config.toml >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_duplicate_resolution_error_uk_multiple_files_multicol_index.94783.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files_multicol_index/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files_multicol_index/data --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_resolution_error_uk_multiple_files_multicol_index.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files_multicol_index/config.toml
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

tidb lightning encountered error: [Lightning:Restore:ErrFoundIndexConflictRecords]found index conflict records in table a, index name is 'a.key_bd', unique key is '[101 9]', primary key is '1'
warning: GOCOVERDIR not set, no coverage data emitted
+ '[' 0 -eq 0 ']'
+ tail -n 10 /tmp/lightning_test/lightning.log
+ grep ERROR
+ tail -n 1
+ grep -Fq '[Lightning:Restore:ErrFoundIndexConflictRecords]found index conflict records in table a, index name is '\''a.key_bd'\'', unique key is '\''[101 9]'\'', primary key is '\''1'\'''
+ check_not_contains 'the whole procedure completed' /tmp/lightning_test/lightning.log
*===== TEST: [lightning_duplicate_resolution_error_uk_multiple_files_multicol_index] success! =====*
Starting PD...
Run cases: lightning_duplicate_resolution_incremental
Starting PD...
{
  "version": "v8.2.0-alpha-15-gf83febabe",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
{
  "version": "v8.2.0-alpha-15-gf83febabe",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-86-gb421b729c8-dirty","git_hash":"b421b729c8a7c43ad7250f38e655cdda03d0ca77"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }, "engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-86-gb421b729c8-dirty","git_hash":"b421b729c8a7c43ad7250f38e655cdda03d0ca77"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
TiFlash started.
selected test cases: lightning_s3
TiFlash started.
selected test cases: lightning_duplicate_resolution_incremental
*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_s3/run.sh... =====*
+ DB=s3_test
+ TABLE=tbl
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_s3/run.sh
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_s3
++ pwd
+ CUR=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_s3
+ check_cluster_version 4 0 0 'local backend'
+ set -euE
+ [[ -z /tmp/lightning_test ]]
+ DBPATH=/tmp/lightning_test/s3.mydump
+ export MINIO_ACCESS_KEY=s3accesskey
+ MINIO_ACCESS_KEY=s3accesskey
+ export MINIO_SECRET_KEY=s3secretkey
+ MINIO_SECRET_KEY=s3secretkey
+ export MINIO_BROWSER=off
+ MINIO_BROWSER=off
+ export S3_ENDPOINT=127.0.0.1:9900
+ S3_ENDPOINT=127.0.0.1:9900
+ rm -rf /tmp/lightning_test/s3_test
+ mkdir -p /tmp/lightning_test/s3_test
+ i=0
+ curl -o /dev/null -v -s http://127.0.0.1:9900/
+ minio server --address 127.0.0.1:9900 /tmp/lightning_test/s3.mydump
* About to connect() to 127.0.0.1 port 9900 (#0)
*   Trying 127.0.0.1...
* Connection refused
* Failed connect to 127.0.0.1:9900; Connection refused
* Closing connection 0
+ i=1
+ '[' 1 -gt 30 ']'
+ sleep 2
*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental/run.sh... =====*
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental/run.sh
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental
++ pwd
+ CUR=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental
+ check_cluster_version 5 2 0 'duplicate detection'
+ LOG_FILE1=/tmp/lightning_test/lightning-duplicate-resolution1.log
+ LOG_FILE2=/tmp/lightning_test/lightning-duplicate-resolution2.log
+ export 'GO_FAILPOINTS=github.com/pingcap/tidb/lightning/pkg/importer/SlowDownCheckDupe=return(10)'
+ GO_FAILPOINTS='github.com/pingcap/tidb/lightning/pkg/importer/SlowDownCheckDupe=return(10)'
+ counter=0
+ '[' 0 -lt 10 ']'
+ grep -Fq 'start to sleep several seconds before checking other dupe' /tmp/lightning_test/lightning-duplicate-resolution1.log
+ run_lightning --backend local --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_resolution_incremental.sorted1 --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-duplicate-resolution1.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental/config1.toml
grep: /tmp/lightning_test/lightning-duplicate-resolution1.log: No such file or directory
+ (( counter += 1 ))
+ echo 'waiting for lightning 1 starts'
waiting for lightning 1 starts
+ sleep 1
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:41:42 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental --backend' local --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_resolution_incremental.sorted1 --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-duplicate-resolution1.log --config '/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental/config1.toml >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_duplicate_resolution_incremental.96623.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental/data --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_resolution_incremental.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_resolution_incremental.sorted1 --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-duplicate-resolution1.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental/config1.toml
Verbose debug logs will be written to /tmp/lightning_test/lightning-duplicate-resolution1.log

+ '[' 1 -lt 10 ']'
+ grep -Fq 'start to sleep several seconds before checking other dupe' /tmp/lightning_test/lightning-duplicate-resolution1.log
+ (( counter += 1 ))
+ echo 'waiting for lightning 1 starts'
waiting for lightning 1 starts
+ sleep 1

 You are running an older version of MinIO released 3 years ago 
 Update: Run `mc admin update` 


Attempting encryption of all config, IAM users and policies on MinIO backend
Endpoint: http://127.0.0.1:9900 

Object API (Amazon S3 compatible):
   Go:         https://docs.min.io/docs/golang-client-quickstart-guide
   Java:       https://docs.min.io/docs/java-client-quickstart-guide
   Python:     https://docs.min.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
   .NET:       https://docs.min.io/docs/dotnet-client-quickstart-guide
+ curl -o /dev/null -v -s http://127.0.0.1:9900/
* About to connect() to 127.0.0.1 port 9900 (#0)
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 9900 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:9900
> Accept: */*
> 
< HTTP/1.1 403 Forbidden
< Accept-Ranges: bytes
< Content-Length: 226
< Content-Security-Policy: block-all-mixed-content
< Content-Type: application/xml
< Server: MinIO
< Vary: Origin
< X-Amz-Request-Id: 17CCE6C8002DF006
< X-Xss-Protection: 1; mode=block
< Date: Mon, 06 May 2024 12:41:44 GMT
< 
{ [data not shown]
* Connection #0 to host 127.0.0.1 left intact
+ BUCKET=test-bucket
+ DATA_PATH=/tmp/lightning_test/s3.mydump/test-bucket
+ mkdir -p /tmp/lightning_test/s3.mydump/test-bucket
+ echo 'CREATE DATABASE s3_test;'
+ echo 'CREATE TABLE t(i INT, s varchar(32));'
+ echo 'INSERT INTO tbl (i, s) VALUES (1, "1"),(2, "test2"), (3, "qqqtest");'
+ cat
+ final_ret_code=0
+ test_import_non_existing_path
+ rm -f /tmp/lightning_test/lightning.log
+ local 'SOURCE_DIR=s3://test-bucket/not-exist-path?endpoint=http%3A//127.0.0.1%3A9900&access_key=s3accesskey&secret_access_key=s3secretkey&force_path_style=true'
+ run_lightning -d 's3://test-bucket/not-exist-path?endpoint=http%3A//127.0.0.1%3A9900&access_key=s3accesskey&secret_access_key=s3secretkey&force_path_style=true' --backend local
+ '[' 2 -lt 10 ']'
+ grep -Fq 'start to sleep several seconds before checking other dupe' /tmp/lightning_test/lightning-duplicate-resolution1.log
+ echo 'lightning 1 already starts waiting for dupe'
lightning 1 already starts waiting for dupe
+ break
+ '[' 2 -ge 10 ']'
+ wait
+ run_lightning --backend local --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_resolution_incremental.sorted2 --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-duplicate-resolution2.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental/config2.toml
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:41:44 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental --backend' local --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_resolution_incremental.sorted2 --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-duplicate-resolution2.log --config '/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental/config2.toml >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_duplicate_resolution_incremental.96676.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental/data --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_resolution_incremental.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_resolution_incremental.sorted2 --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-duplicate-resolution2.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental/config2.toml
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

+ grep -Eq 'data-source-dir .* doesn'\''t exist or contains no files' /tmp/lightning_test/lightning.log
+ return 0
+ test_import_empty_dir
+ rm -f /tmp/lightning_test/lightning.log
+ local emptyPath=empty-bucket/empty-path
+ mkdir -p /tmp/lightning_test/s3.mydump/empty-bucket/empty-path
+ local 'SOURCE_DIR=s3://empty-bucket/empty-path/not-exist-path?endpoint=http%3A//127.0.0.1%3A9900&access_key=s3accesskey&secret_access_key=s3secretkey&force_path_style=true'
+ run_lightning -d 's3://empty-bucket/empty-path/not-exist-path?endpoint=http%3A//127.0.0.1%3A9900&access_key=s3accesskey&secret_access_key=s3secretkey&force_path_style=true' --backend local
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

+ grep -Eq 'data-source-dir .* doesn'\''t exist or contains no files' /tmp/lightning_test/lightning.log
+ return 0
+ test_normal_import
+ rm -f /tmp/lightning_test/lightning.log
+ cleanup_db_and_table
+ run_sql 'DROP DATABASE IF EXISTS s3_test;'
+ run_sql 'DROP TABLE IF EXISTS s3_test.tbl;'
+ local 'SOURCE_DIR=s3://test-bucket/?endpoint=http%3A//127.0.0.1%3A9900&access_key=s3accesskey&secret_access_key=s3secretkey&force_path_style=true'
+ run_lightning -d 's3://test-bucket/?endpoint=http%3A//127.0.0.1%3A9900&access_key=s3accesskey&secret_access_key=s3secretkey&force_path_style=true' --backend local
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

Verbose debug logs will be written to /tmp/lightning_test/lightning-duplicate-resolution2.log

tidb lightning exit successfully
PASS
coverage: 24.9% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ run_sql 'SELECT count(*), sum(i) FROM `s3_test`.tbl'
*************************** 1. row ***************************
count(*): 7
  sum(i): 413
+ check_contains 'count(*): 7'
+ check_contains 'sum(i): 413'
+ return 0
+ test_import_with_checkpoint
+ rm -f /tmp/lightning_test/lightning.log
+ cleanup_db_and_table
+ run_sql 'DROP DATABASE IF EXISTS s3_test;'
+ run_sql 'DROP TABLE IF EXISTS s3_test.tbl;'
+ local 'SOURCE_DIR=s3://test-bucket/?endpoint=http%3A//127.0.0.1%3A9900&access_key=s3accesskey&secret_access_key=s3secretkey&force_path_style=true'
+ run_lightning -d 's3://test-bucket/?endpoint=http%3A//127.0.0.1%3A9900&access_key=s3accesskey&secret_access_key=s3secretkey&force_path_style=true' --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_s3/config_s3_checkpoint.toml
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

tidb lightning exit successfully
PASS
coverage: 25.1% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ run_sql 'SELECT count(*), sum(i) FROM `s3_test`.tbl'
*************************** 1. row ***************************
count(*): 7
  sum(i): 413
+ check_contains 'count(*): 7'
+ check_contains 'sum(i): 413'
+ test_import_using_manual_path_config
+ rm -f /tmp/lightning_test/lightning.log
+ cleanup_db_and_table
+ run_sql 'DROP DATABASE IF EXISTS s3_test;'
+ run_sql 'DROP TABLE IF EXISTS s3_test.tbl;'
+ local bucket_02=test-bucket-02
+ local sub_path_02=to-be-imported
+ local base_path_02=/tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported
+ rm -rf /tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported
+ mkdir -p /tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported
+ touch /tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported/s3_test-schema-create.sql
+ cp /tmp/lightning_test/s3.mydump/test-bucket/s3_test.tbl-schema.sql /tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported/
+ local sql_data_path_02=/tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported/data-sql
+ local csv_data_path_02=/tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported/data-csv
+ local parquet_data_path_02=/tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported/data-parquet
+ mkdir -p /tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported/data-sql/empty-dir
+ cp /tmp/lightning_test/s3.mydump/test-bucket/s3_test.tbl.sql /tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported/data-sql/
+ touch /tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported/data-sql/dummy.sql
+ mkdir -p /tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported/data-csv/empty-dir
+ cp /tmp/lightning_test/s3.mydump/test-bucket/s3_test.tbl.0.csv /tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported/data-csv/
+ touch /tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported/data-csv/dummy.csv
+ mkdir -p /tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported/data-parquet/empty-dir
+ local 'SOURCE_DIR=s3://test-bucket-02/to-be-imported?endpoint=http%3A//127.0.0.1%3A9900&access_key=s3accesskey&secret_access_key=s3secretkey&force_path_style=true'
+ run_lightning -d 's3://test-bucket-02/to-be-imported?endpoint=http%3A//127.0.0.1%3A9900&access_key=s3accesskey&secret_access_key=s3secretkey&force_path_style=true' --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_s3/config_manual_files.toml
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

tidb lightning exit successfully
PASS
coverage: 24.9% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ run_sql 'SELECT count(*), sum(i) FROM `s3_test`.tbl'
*************************** 1. row ***************************
count(*): 7
  sum(i): 413
+ check_contains 'count(*): 7'
+ check_contains 'sum(i): 413'
+ return 0
+ exit 0
*===== TEST: [lightning_s3] success! =====*
tidb lightning exit successfully
PASS
coverage: 13.7% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ '[' 130 == 130 ']'
+ '[' 12548524131860600223 == 12548524131860600223 ']'
+ '[' 30 == 30 ']'
+ '[' 3322292718850045174 == 3322292718850045174 ']'
*===== TEST: [lightning_add_index] success! =====*
[Pipeline] }
Cache not saved (ws/jenkins-pingcap-tidb-merged_integration_lightning_test-538/lightning-tests already exists)
[Pipeline] // cache
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
Run cases: lightning_alter_random
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Starting PD...

Import Data Error Summary: 
+---+---------------------+-------------+--------------------------------------------+
| # | ERROR TYPE          | ERROR COUNT | ERROR DATA TABLE                           |
+---+---------------------+-------------+--------------------------------------------+
| 1 | Unique Key Conflict |          10 | `lightning_task_info_dupe_resolve_incremen |
|   |                     |             | tal`.`conflict_error_v3`                   |
+---+---------------------+-------------+--------------------------------------------+

tidb lightning exit successfully
PASS
coverage: 27.2% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
tidb lightning exit successfully
PASS
coverage: 28.3% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ export GO_FAILPOINTS=
+ GO_FAILPOINTS=
+ run_sql 'admin check table dup_resolve_detect.ta'
+ run_sql 'select count(*), sum(id) from dup_resolve_detect.ta where id < 100'
*************************** 1. row ***************************
count(*): 15
 sum(id): 120
+ check_contains 'count(*): 15'
+ check_contains 'sum(id): 120'
+ run_sql 'select count(*), sum(id) from dup_resolve_detect.ta where id > 100'
*************************** 1. row ***************************
count(*): 16
 sum(id): 1896
+ check_contains 'count(*): 16'
+ check_contains 'sum(id): 1896'
*===== TEST: [lightning_duplicate_resolution_incremental] success! =====*
[Pipeline] }
Cache not saved (ws/jenkins-pingcap-tidb-merged_integration_lightning_test-538/lightning-tests already exists)
[Pipeline] // cache
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
{
  "version": "v8.2.0-alpha-15-gf83febabe",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-86-gb421b729c8-dirty","git_hash":"b421b729c8a7c43ad7250f38e655cdda03d0ca77"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }, "log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
TiFlash started.
selected test cases: lightning_alter_random
*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_alter_random/run.sh... =====*
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:42:15 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_alter_random --backend' 'tidb >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_alter_random.25466.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_alter_random/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_alter_random/data --sorted-kv-dir /tmp/lightning_test/lightning_alter_random.sorted --enable-checkpoint=0 --check-requirements=0 --backend tidb
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

tidb lightning exit successfully
PASS
coverage: 14.7% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
*************************** 1. row ***************************
count(*): 3
*************************** 1. row ***************************
inc: 1
*************************** 2. row ***************************
inc: 3
*************************** 3. row ***************************
inc: 2
*************************** 1. row ***************************
inc: 30002
*************************** 2. row ***************************
inc: 1
*************************** 3. row ***************************
inc: 3
*************************** 4. row ***************************
inc: 2
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:42:15 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_alter_random --backend' 'local >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_alter_random.25535.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_alter_random/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_alter_random/data --sorted-kv-dir /tmp/lightning_test/lightning_alter_random.sorted --enable-checkpoint=0 --check-requirements=0 --backend local
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

tidb lightning exit successfully
PASS
coverage: 22.9% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
*************************** 1. row ***************************
count(*): 3
*************************** 1. row ***************************
inc: 1
*************************** 2. row ***************************
inc: 3
*************************** 3. row ***************************
inc: 2
*************************** 1. row ***************************
inc: 1
*************************** 2. row ***************************
inc: 3
*************************** 3. row ***************************
inc: 2
*************************** 4. row ***************************
inc: 4
*===== TEST: [lightning_alter_random] success! =====*
Run cases: lightning_auto_columns
Starting PD...
{
  "version": "v8.2.0-alpha-15-gf83febabe",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-86-gb421b729c8-dirty","git_hash":"b421b729c8a7c43ad7250f38e655cdda03d0ca77"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }, "log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
TiFlash started.
selected test cases: lightning_auto_columns
*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_auto_columns/run.sh... =====*
+ run_sql 'DROP DATABASE IF EXISTS lightning_auto_cols;'
+ run_lightning
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 20:42:40 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_auto_columns  >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_auto_columns.27405.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_auto_columns/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_auto_columns/data --sorted-kv-dir /tmp/lightning_test/lightning_auto_columns.sorted --enable-checkpoint=0 --check-requirements=0
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

tidb lightning exit successfully
PASS
coverage: 23.5% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ run_sql 'SELECT CONCAT_WS('\'':'\'', id, c) AS row_data FROM lightning_auto_cols.t_auto_incr;'
*************************** 1. row ***************************
row_data: 1:normal_pk_01
*************************** 2. row ***************************
row_data: 2:null_pk_02
*************************** 3. row ***************************
row_data: 3:null_pk_03
*************************** 4. row ***************************
row_data: 4:normal_pk_04
+ check_contains 'row_data: 1:normal_pk_01'
+ check_contains 'row_data: 2:null_pk_02'
+ check_contains 'row_data: 3:null_pk_03'
+ check_contains 'row_data: 4:normal_pk_04'
+ run_sql 'SELECT COUNT(*) AS row_count FROM lightning_auto_cols.t_auto_incr;'
*************************** 1. row ***************************
row_count: 4
+ check_contains 'row_count: 4'
+ run_sql 'SELECT CONCAT_WS('\'':'\'', id, c) AS row_data FROM lightning_auto_cols.t_auto_random;'
*************************** 1. row ***************************
row_data: 1:normal_pk_01
*************************** 2. row ***************************
row_data: 4:normal_pk_04
*************************** 3. row ***************************
row_data: 1152921504606846978:null_pk_02
*************************** 4. row ***************************
row_data: 1152921504606846979:null_pk_03
+ check_contains 'row_data: 1:normal_pk_01'
+ check_contains :null_pk_02
+ check_not_contains 'row_data: 0:null_pk_02'
+ check_contains :null_pk_03
+ check_not_contains 'row_data: 0:null_pk_03'
+ check_contains 'row_data: 4:normal_pk_04'
+ run_sql 'SELECT COUNT(*) AS row_count FROM lightning_auto_cols.t_auto_random;'
*************************** 1. row ***************************
row_count: 4
+ check_contains 'row_count: 4'
*===== TEST: [lightning_auto_columns] success! =====*
[Pipeline] }
Cache not saved (ws/jenkins-pingcap-tidb-merged_integration_lightning_test-538/lightning-tests already exists)
[Pipeline] // cache
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Finished: SUCCESS