Console Output

Skipping 1,017 KB.. Full Log
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

tidb lightning exit successfully
PASS
coverage: 23.2% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ check_result
+ run_sql 'SHOW DATABASES;'
*************************** 1. row ***************************
Database: INFORMATION_SCHEMA
*************************** 2. row ***************************
Database: METRICS_SCHEMA
*************************** 3. row ***************************
Database: PERFORMANCE_SCHEMA
*************************** 4. row ***************************
Database: db
*************************** 5. row ***************************
Database: mysql
*************************** 6. row ***************************
Database: sys
*************************** 7. row ***************************
Database: test
+ check_contains 'Database: db'
+ run_sql 'SHOW TABLES IN db;'
*************************** 1. row ***************************
Tables_in_db: test
*************************** 2. row ***************************
Tables_in_db: test1
+ check_contains 'Tables_in_db: test'
+ check_contains 'Tables_in_db: test1'
+ run_sql 'SELECT count(*) FROM db.test;'
*************************** 1. row ***************************
count(*): 2
+ check_contains 'count(*): 2'
+ run_sql 'SELECT count(*) FROM db.test1;'
*************************** 1. row ***************************
count(*): 2
+ check_contains 'count(*): 2'
+ run_sql 'INSERT INTO db.test(b) VALUES(11);'
+ run_sql 'INSERT INTO db.test1(b) VALUES(22);'
+ tee -a /tmp/lightning_test/sql_res.lightning_max_incr.txt
ERROR 1467 (HY000) at line 1: Failed to read auto-increment value from storage engine
+ check_contains ERROR
+ cleanup
+ rm -f /tmp/lightning_test/lightning.log
+ rm -rf /tmp/lightning_test/sst
+ run_sql 'DROP DATABASE IF EXISTS db;'
*===== TEST: [lightning_max_incr] success! =====*
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
tidb lightning exit successfully
PASS
coverage: 23.9% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
*************************** 1. row ***************************
count(*): 4
  sum(c): 10
*************************** 1. row ***************************
count(*): 4
  sum(c): 46
[2024/05/06 15:01:43.888 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc002c34000/127.0.0.1:2379] [attempt=0] [error="rpc error: code = Canceled desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""]
[2024/05/06 15:01:43.888 +08:00] [WARN] [retry_interceptor.go:62] ["retrying of unary invoker failed"] [target=etcd-endpoints://0xc002be65a0/127.0.0.1:2379] [attempt=0] [error="rpc error: code = Canceled desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""]
/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_services: line 54: 19332 Killed                  pd-server --join "https://$PD_ADDR" --client-urls "https://${PD_ADDR}2" --peer-urls "https://${PD_PEER_ADDR}2" --log-file "$TEST_DIR/pd2.log" --data-dir "$TEST_DIR/pd2" --name pd2 --config $PD_CONFIG
/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_services: line 54: 19382 Killed                  pd-server --join "https://$PD_ADDR" --client-urls "https://${PD_ADDR}3" --peer-urls "https://${PD_PEER_ADDR}3" --log-file "$TEST_DIR/pd3.log" --data-dir "$TEST_DIR/pd3" --name pd3 --config $PD_CONFIG
/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_services: line 54: 19429 Killed                  tidb-server -P 4000 --status 10080 --advertise-address="127.0.0.1" --store tikv --path "$PD_ADDR" --config "$TIDB_CONFIG" --log-file "$TEST_DIR/tidb.log"
delete /tmp/lightning_test/tiflash
delete /tmp/lightning_test/tikv1
delete /tmp/lightning_test/tikv3
delete /tmp/lightning_test/tikv2
delete /tmp/lightning_test/pd3
delete /tmp/lightning_test/pd
delete /tmp/lightning_test/pd2
Starting PD...
Starting PD...
{
  "version": "v8.2.0-alpha-14-g1679dbca2",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
[Pipeline] }
Cache not saved (ws/jenkins-pingcap-tidb-merged_integration_lightning_test-533/lightning-tests already exists)
[Pipeline] // cache
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
{
  "version": "v8.2.0-alpha-14-g1679dbca2",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-81-g516977e1f9-dirty","git_hash":"516977e1f99d9df87e142e1ea280baed56a12a70"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }, "engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }, "log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
TiFlash started.
selected test cases: lightning_restore
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_restore/run.sh... =====*
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 15:02:01 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_restore -d' '/tmp/lightning_test/restore.mydump >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_restore.23435.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_restore/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_restore/data --sorted-kv-dir /tmp/lightning_test/lightning_restore.sorted --enable-checkpoint=0 --check-requirements=0 -d /tmp/lightning_test/restore.mydump
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-81-g516977e1f9-dirty","git_hash":"516977e1f99d9df87e142e1ea280baed56a12a70"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }, "engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
TiFlash started.
*===== TEST: [lightning_pd_leader_switch] success! =====*
{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-81-g516977e1f9-dirty","git_hash":"516977e1f99d9df87e142e1ea280baed56a12a70"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }, "engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
tidb lightning exit successfully
PASS
coverage: 22.8% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
Import finished
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
Run cases: lightning_add_index
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 15:02:07 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_restore  >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_restore.23755.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_restore/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_restore/data --sorted-kv-dir /tmp/lightning_test/lightning_restore.sorted --enable-checkpoint=0 --check-requirements=0
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

TiFlash started.
selected test cases: lightning_duplicate_detection_new
Starting PD...
*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/run.sh... =====*
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/run.sh
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new
++ pwd
+ CUR=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new
+ LOG_FILE=/tmp/lightning_test/lightning-duplicate-detection-new.log
+ cleanup
+ run_sql 'DROP DATABASE IF EXISTS test'
+ run_sql 'DROP DATABASE IF EXISTS lightning_task_info'
+ rm -f /tmp/lightning_test/lightning-duplicate-detection-new.log
+ run_lightning --backend tidb --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/tidb-replace.toml --log-file /tmp/lightning_test/lightning-duplicate-detection-new.log
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 15:02:10 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new --backend' tidb --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/tidb-replace.toml --log-file '/tmp/lightning_test/lightning-duplicate-detection-new.log >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_duplicate_detection_new.86528.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/data --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_detection_new.sorted --enable-checkpoint=0 --check-requirements=0 --backend tidb --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/tidb-replace.toml --log-file /tmp/lightning_test/lightning-duplicate-detection-new.log
Verbose debug logs will be written to /tmp/lightning_test/lightning-duplicate-detection-new.log


Import Data Error Summary: 
+---+---------------------+-------------+------------------------------------------+
| # | ERROR TYPE          | ERROR COUNT | ERROR DATA TABLE                         |
+---+---------------------+-------------+------------------------------------------+
|[31m 1 [0m|[31m Unique Key Conflict [0m|[31m         227 [0m|[31m `lightning_task_info`.`conflict_records` [0m|
+---+---------------------+-------------+------------------------------------------+

tidb lightning exit successfully
PASS
coverage: 15.0% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
++ run_sql 'SELECT count(*) AS total_count FROM test.dup_detect'
++ grep total_count
++ awk '{print $2}'
+ expected_rows=174
++ run_sql 'SELECT group_concat(col1 ORDER BY col1) AS pks FROM test.dup_detect'
++ grep pks
++ awk '{print $2}'
+ expected_pks=1,2,3,4,5,6,7,8,9,11,12,13,14,15,16,17,19,20,21,22,23,24,25,26,27,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,46,47,48,49,50,51,52,53,54,55,56,57,58,59,61,62,63,65,66,67,68,69,70,72,74,75,76,77,78,80,81,83,86,87,88,89,90,91,92,93,94,95,96,98,99,100,101,102,103,104,106,107,108,110,111,112,113,114,115,116,117,118,120,121,122,123,124,125,126,127,128,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,155,157,158,159,160,161,162,164,165,166,167,168,169,170,171,172,174,175,176,177,178,179,180,181,182,183,185,186,188,189,190,192,193,194,195,196,197,199,987
+ cleanup
+ run_sql 'DROP DATABASE IF EXISTS test'
+ run_sql 'DROP DATABASE IF EXISTS lightning_task_info'
+ rm -f /tmp/lightning_test/lightning-duplicate-detection-new.log
+ run_lightning --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/local-replace.toml --log-file /tmp/lightning_test/lightning-duplicate-detection-new.log
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 15:02:11 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new --backend' local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/local-replace.toml --log-file '/tmp/lightning_test/lightning-duplicate-detection-new.log >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_duplicate_detection_new.86582.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/data --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_detection_new.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/local-replace.toml --log-file /tmp/lightning_test/lightning-duplicate-detection-new.log
Verbose debug logs will be written to /tmp/lightning_test/lightning-duplicate-detection-new.log


Import Data Error Summary: 
+---+---------------------+-------------+---------------------------------------+
| # | ERROR TYPE          | ERROR COUNT | ERROR DATA TABLE                      |
+---+---------------------+-------------+---------------------------------------+
|[31m 1 [0m|[31m Unique Key Conflict [0m|[31m         227 [0m|[31m `lightning_task_info`.`conflict_view` [0m|
+---+---------------------+-------------+---------------------------------------+

tidb lightning exit successfully
PASS
coverage: 28.0% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
++ run_sql 'SELECT count(*) AS total_count FROM test.dup_detect'
++ grep total_count
++ awk '{print $2}'
+ actual_rows=174
++ run_sql 'SELECT group_concat(col1 ORDER BY col1) AS pks FROM test.dup_detect'
++ grep pks
++ awk '{print $2}'
+ actual_pks=1,2,3,4,5,6,7,8,9,11,12,13,14,15,16,17,19,20,21,22,23,24,25,26,27,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,46,47,48,49,50,51,52,53,54,55,56,57,58,59,61,62,63,65,66,67,68,69,70,72,74,75,76,77,78,80,81,83,86,87,88,89,90,91,92,93,94,95,96,98,99,100,101,102,103,104,106,107,108,110,111,112,113,114,115,116,117,118,120,121,122,123,124,125,126,127,128,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,155,157,158,159,160,161,162,164,165,166,167,168,169,170,171,172,174,175,176,177,178,179,180,181,182,183,185,186,188,189,190,192,193,194,195,196,197,199,987
+ '[' 174 '!=' 174 ']'
+ '[' 1,2,3,4,5,6,7,8,9,11,12,13,14,15,16,17,19,20,21,22,23,24,25,26,27,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,46,47,48,49,50,51,52,53,54,55,56,57,58,59,61,62,63,65,66,67,68,69,70,72,74,75,76,77,78,80,81,83,86,87,88,89,90,91,92,93,94,95,96,98,99,100,101,102,103,104,106,107,108,110,111,112,113,114,115,116,117,118,120,121,122,123,124,125,126,127,128,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,155,157,158,159,160,161,162,164,165,166,167,168,169,170,171,172,174,175,176,177,178,179,180,181,182,183,185,186,188,189,190,192,193,194,195,196,197,199,987 '!=' 1,2,3,4,5,6,7,8,9,11,12,13,14,15,16,17,19,20,21,22,23,24,25,26,27,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,46,47,48,49,50,51,52,53,54,55,56,57,58,59,61,62,63,65,66,67,68,69,70,72,74,75,76,77,78,80,81,83,86,87,88,89,90,91,92,93,94,95,96,98,99,100,101,102,103,104,106,107,108,110,111,112,113,114,115,116,117,118,120,121,122,123,124,125,126,127,128,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,155,157,158,159,160,161,162,164,165,166,167,168,169,170,171,172,174,175,176,177,178,179,180,181,182,183,185,186,188,189,190,192,193,194,195,196,197,199,987 ']'
+ run_sql 'SELECT count(*) FROM lightning_task_info.conflict_records'
*************************** 1. row ***************************
count(*): 227
+ check_contains 'count(*): 227'
+ run_sql 'SELECT count(*) FROM lightning_task_info.conflict_records WHERE error = '\'''\'''
*************************** 1. row ***************************
count(*): 0
+ check_contains 'count(*): 0'
+ run_sql 'SELECT * FROM lightning_task_info.conflict_records WHERE row_id = 12'
*************************** 1. row ***************************
    task_id: 1714978931552044289
create_time: 2024-05-06 15:02:12.290493
 table_name: `test`.`dup_detect`
       path: test.dup_detect.1.sql
     offset: 797
      error: [kv:1062]Duplicate entry '171' for key 'dup_detect.PRIMARY'
     row_id: 12
   row_data: (171,'yRxZE',9201592769833450947,'xs3d',5,4,283270321)
+ check_contains '(171,'\''yRxZE'\'',9201592769833450947,'\''xs3d'\'',5,4,283270321)'
+ check_contains '[kv:1062]Duplicate entry '\''171'\'' for key '\''dup_detect.PRIMARY'\'''
+ run_sql 'SELECT * FROM lightning_task_info.conflict_records WHERE row_id = 1'
*************************** 1. row ***************************
    task_id: 1714978931552044289
create_time: 2024-05-06 15:02:12.274260
 table_name: `test`.`dup_detect`
       path: test.dup_detect.1.sql
     offset: 90
      error: [kv:1062]Duplicate entry '0-177543185' for key 'dup_detect.uniq_col6_col7'
     row_id: 1
   row_data: (87,'nEoKu',7836621565948506759,'y6',48,0,177543185)
+ check_contains '(87,'\''nEoKu'\'',7836621565948506759,'\''y6'\'',48,0,177543185)'
+ check_contains '[kv:1062]Duplicate entry '\''0-177543185'\'' for key '\''dup_detect.uniq_col6_col7'\'''
+ cleanup
+ run_sql 'DROP DATABASE IF EXISTS test'
{
  "version": "v8.2.0-alpha-14-g1679dbca2",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
+ run_sql 'DROP DATABASE IF EXISTS lightning_task_info'
+ rm -f /tmp/lightning_test/lightning-duplicate-detection-new.log
+ run_lightning --backend tidb --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/tidb-ignore.toml --log-file /tmp/lightning_test/lightning-duplicate-detection-new.log
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 15:02:13 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new --backend' tidb --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/tidb-ignore.toml --log-file '/tmp/lightning_test/lightning-duplicate-detection-new.log >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_duplicate_detection_new.86691.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/data --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_detection_new.sorted --enable-checkpoint=0 --check-requirements=0 --backend tidb --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/tidb-ignore.toml --log-file /tmp/lightning_test/lightning-duplicate-detection-new.log
Verbose debug logs will be written to /tmp/lightning_test/lightning-duplicate-detection-new.log


Import Data Error Summary: 
+---+---------------------+-------------+------------------------------------------+
| # | ERROR TYPE          | ERROR COUNT | ERROR DATA TABLE                         |
+---+---------------------+-------------+------------------------------------------+
|[31m 1 [0m|[31m Unique Key Conflict [0m|[31m         228 [0m|[31m `lightning_task_info`.`conflict_records` [0m|
+---+---------------------+-------------+------------------------------------------+

tidb lightning exit successfully
PASS
coverage: 15.2% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
++ run_sql 'SELECT count(*) AS total_count FROM test.dup_detect'
++ grep total_count
++ awk '{print $2}'
+ expected_rows=173
++ run_sql 'SELECT group_concat(col1 ORDER BY col1) AS pks FROM test.dup_detect'
++ grep pks
++ awk '{print $2}'
+ expected_pks=1,2,3,4,5,6,7,8,9,11,12,13,14,15,16,17,19,20,21,22,23,24,25,26,27,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,46,47,48,49,50,51,52,53,54,55,56,57,58,59,61,62,63,65,66,67,68,69,70,72,74,75,76,77,78,80,81,83,86,87,88,89,90,91,92,93,94,95,96,98,99,100,101,102,103,104,106,107,108,110,111,112,113,114,115,116,117,118,120,121,122,123,124,125,126,127,128,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,155,157,158,159,160,161,162,164,165,166,167,168,169,170,171,172,174,175,176,177,178,179,180,181,182,183,185,186,188,189,190,192,193,194,195,196,197,199
+ cleanup
+ run_sql 'DROP DATABASE IF EXISTS test'
+ run_sql 'DROP DATABASE IF EXISTS lightning_task_info'
+ rm -f /tmp/lightning_test/lightning-duplicate-detection-new.log
+ run_lightning --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/local-error.toml --log-file /tmp/lightning_test/lightning-duplicate-detection-new.log
+ grep -q 'duplicate key in table `test`.`dup_detect` caused by index .*, but because checkpoint is off we can'\''t have more details'
+ grep -q 'duplicate key in table `test`.`dup_detect` caused by index .*, but because checkpoint is off we can'\''t have more details' /tmp/lightning_test/lightning-duplicate-detection-new.log
+ run_sql 'SELECT * FROM lightning_task_info.conflict_records'
*************************** 1. row ***************************
    task_id: 1714978935100136238
create_time: 2024-05-06 15:02:15.712549
 table_name: `test`.`dup_detect`
       path: <unknown-path>
     offset: -1
      error: duplicate key in table `test`.`dup_detect` caused by index `uniq_col6_col7`, but because checkpoint is off we can't have more details
     row_id: -1
   row_data: <unknown-data>
+ check_contains 'error: duplicate key in table `test`.`dup_detect`'
+ run_lightning --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/local-error.toml --log-file /tmp/lightning_test/lightning-duplicate-detection-new.log --enable-checkpoint=1
+ grep -q 'duplicate entry for key '\''uniq_col6_col7'\'', a pair of conflicting rows are (row 1 counting from offset 0 in file test.dup_detect.1.sql, row 101 counting from offset 0 in file test.dup_detect.4.sql)'
+ grep -q 'duplicate entry for key '\''uniq_col6_col7'\'', a pair of conflicting rows are (row 1 counting from offset 0 in file test.dup_detect.1.sql, row 101 counting from offset 0 in file test.dup_detect.4.sql)' /tmp/lightning_test/lightning-duplicate-detection-new.log
+ run_sql 'SELECT * FROM lightning_task_info.conflict_records'
*************************** 1. row ***************************
    task_id: 1714978935100136238
create_time: 2024-05-06 15:02:15.712549
 table_name: `test`.`dup_detect`
       path: <unknown-path>
     offset: -1
      error: duplicate key in table `test`.`dup_detect` caused by index `uniq_col6_col7`, but because checkpoint is off we can't have more details
     row_id: -1
   row_data: <unknown-data>
*************************** 2. row ***************************
    task_id: 1714978935814471758
create_time: 2024-05-06 15:02:16.499582
 table_name: `test`.`dup_detect`
       path: test.dup_detect.4.sql
     offset: -1
      error: duplicate entry for key 'uniq_col6_col7', a pair of conflicting rows are (row 1 counting from offset 0 in file test.dup_detect.1.sql, row 101 counting from offset 0 in file test.dup_detect.4.sql)
     row_id: 2256
   row_data: <unknown-data>
+ check_contains 'error: duplicate entry for key '\''uniq_col6_col7'\'', a pair of conflicting rows are'
+ check_contains 'restore table `test`.`dup_detect` failed: duplicate entry for key '\''uniq_col6_col7'\'', a pair of conflicting rows are (row 1 counting from offset 0 in file test.dup_detect.1.sql, row 101 counting from offset 0 in file test.dup_detect.4.sql)' /tmp/lightning_test/lightning-duplicate-detection-new.log
+ run_lightning_ctl --enable-checkpoint=1 --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/local-error.toml '--checkpoint-error-destroy=`test`.`dup_detect`'
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning_ctl
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
+ tidb-lightning-ctl.test -test.coverprofile=/tmp/group_cover/cov.ctl.lightning_duplicate_detection_new.86866.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/data --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_detection_new.sorted --enable-checkpoint=0 --check-requirements=0 --enable-checkpoint=1 --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/local-error.toml '--checkpoint-error-destroy=`test`.`dup_detect`'
[2024/05/06 15:02:16.601 +08:00] [INFO] [pd_service_discovery.go:1016] ["[pd] switch leader"] [new-leader=https://127.0.0.1:2379] [old-leader=]
[2024/05/06 15:02:16.601 +08:00] [INFO] [pd_service_discovery.go:498] ["[pd] init cluster id"] [cluster-id=7365778342334910510]
Dropping table: `test`.`dup_detect`
Closing and cleaning up engine: `test`.`dup_detect` -1
Closing and cleaning up engine: `test`.`dup_detect` 0
[2024/05/06 15:02:16.838 +08:00] [INFO] [pd_service_discovery.go:637] ["[pd] close pd service discovery client"]
[2024/05/06 15:02:16.838 +08:00] [INFO] [pd_service_discovery.go:550] ["[pd] exit member loop due to context canceled"]
[2024/05/06 15:02:16.839 +08:00] [INFO] [client.go:329] ["[pd] http client closed"] [source=lightning-ctl]
PASS
coverage: 2.9% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
++ ls /tmp/lightning_test/lightning_duplicate_detection_new.sorted
++ wc -l
+ files_left=0
+ '[' 0 -ne 0 ']'
+ rm -rf /tmp/lightning_test/lightning_duplicate_detection_new.sorted
+ cleanup
+ run_sql 'DROP DATABASE IF EXISTS test'
+ run_sql 'DROP DATABASE IF EXISTS lightning_task_info'
+ rm -f /tmp/lightning_test/lightning-duplicate-detection-new.log
+ export 'GO_FAILPOINTS=github.com/pingcap/tidb/lightning/pkg/importer/FailAfterDuplicateDetection=return()'
+ GO_FAILPOINTS='github.com/pingcap/tidb/lightning/pkg/importer/FailAfterDuplicateDetection=return()'
+ run_lightning --enable-checkpoint=1 --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/local-replace.toml --log-file /tmp/lightning_test/lightning-duplicate-detection-new.log
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 15:02:17 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new --enable-checkpoint=1' --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/local-replace.toml --log-file '/tmp/lightning_test/lightning-duplicate-detection-new.log >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_duplicate_detection_new.86918.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/data --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_detection_new.sorted --enable-checkpoint=0 --check-requirements=0 --enable-checkpoint=1 --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/local-replace.toml --log-file /tmp/lightning_test/lightning-duplicate-detection-new.log
tidb lightning exit successfully
PASS
coverage: 25.4% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
Import finished
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*************************** 1. row ***************************
sum(i): 1
*===== TEST: [lightning_restore] success! =====*
Verbose debug logs will be written to /tmp/lightning_test/lightning-duplicate-detection-new.log

panic: forcing failure after duplicate detection

goroutine 565 [running]:
github.com/pingcap/tidb/lightning/pkg/importer.(*TableImporter).importTable(0xc001343540, {0x64fc8e8, 0xc0012b28a0}, 0xc000b99200, 0xc001d89090)
	/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/pkg/importer/table_import.go:239 +0x1271
github.com/pingcap/tidb/lightning/pkg/importer.(*Controller).importTables.func6()
	/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/pkg/importer/import.go:1437 +0x172
created by github.com/pingcap/tidb/lightning/pkg/importer.(*Controller).importTables in goroutine 59
	/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/pkg/importer/import.go:1432 +0x9dc
+ true
+ unset GO_FAILPOINTS
+ rm -f /tmp/lightning_test/lightning-duplicate-detection-new.log
+ run_lightning_ctl --enable-checkpoint=1 --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/local-replace.toml '--checkpoint-error-ignore=`test`.`dup_detect`'
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning_ctl
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
+ tidb-lightning-ctl.test -test.coverprofile=/tmp/group_cover/cov.ctl.lightning_duplicate_detection_new.86971.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/data --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_detection_new.sorted --enable-checkpoint=0 --check-requirements=0 --enable-checkpoint=1 --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/local-replace.toml '--checkpoint-error-ignore=`test`.`dup_detect`'
[2024/05/06 15:02:18.153 +08:00] [INFO] [pd_service_discovery.go:1016] ["[pd] switch leader"] [new-leader=https://127.0.0.1:2379] [old-leader=]
[2024/05/06 15:02:18.153 +08:00] [INFO] [pd_service_discovery.go:498] ["[pd] init cluster id"] [cluster-id=7365778342334910510]
[2024/05/06 15:02:18.162 +08:00] [INFO] [pd_service_discovery.go:637] ["[pd] close pd service discovery client"]
[2024/05/06 15:02:18.162 +08:00] [INFO] [pd_service_discovery.go:550] ["[pd] exit member loop due to context canceled"]
[2024/05/06 15:02:18.162 +08:00] [INFO] [client.go:329] ["[pd] http client closed"] [source=lightning-ctl]
PASS
coverage: 2.5% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ run_lightning --enable-checkpoint=1 --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/local-replace.toml --log-file /tmp/lightning_test/lightning-duplicate-detection-new.log
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 15:02:18 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new --enable-checkpoint=1' --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/local-replace.toml --log-file '/tmp/lightning_test/lightning-duplicate-detection-new.log >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_duplicate_detection_new.87006.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/data --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_detection_new.sorted --enable-checkpoint=0 --check-requirements=0 --enable-checkpoint=1 --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_detection_new/local-replace.toml --log-file /tmp/lightning_test/lightning-duplicate-detection-new.log
Verbose debug logs will be written to /tmp/lightning_test/lightning-duplicate-detection-new.log

2024/05/06 15:02:18 [JOB 1] WAL file /tmp/lightning_test/lightning_duplicate_detection_new.sorted/duplicates/000002.log with log number 000002 stopped reading at offset: 0; replayed 0 keys in 0 batches
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
Run cases: lightning_routes

Import Data Error Summary: 
+---+---------------------+-------------+---------------------------------------+
| # | ERROR TYPE          | ERROR COUNT | ERROR DATA TABLE                      |
+---+---------------------+-------------+---------------------------------------+
|[31m 1 [0m|[31m Unique Key Conflict [0m|[31m         227 [0m|[31m `lightning_task_info`.`conflict_view` [0m|
+---+---------------------+-------------+---------------------------------------+

tidb lightning exit successfully
PASS
coverage: 27.0% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ run_sql 'SELECT count(*) FROM test.dup_detect'
*************************** 1. row ***************************
count(*): 174
+ check_contains 'count(*): 174'
+ run_sql 'SELECT count(*) FROM lightning_task_info.conflict_records'
*************************** 1. row ***************************
count(*): 227
+ check_contains 'count(*): 227'
+ check_not_contains 'duplicate detection start' /tmp/lightning_test/lightning-duplicate-detection-new.log
*===== TEST: [lightning_duplicate_detection_new] success! =====*
Run cases: lightning_duplicate_resolution_error
{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-81-g516977e1f9-dirty","git_hash":"516977e1f99d9df87e142e1ea280baed56a12a70"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }, "engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
Starting PD...
Starting PD...
TiFlash started.
selected test cases: lightning_add_index
*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/run.sh... =====*
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 15:02:28 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index --config' /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/config1.toml --log-file '/tmp/lightning_test/lightning-add-index1.log >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_add_index.23090.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/data --sorted-kv-dir /tmp/lightning_test/lightning_add_index.sorted --enable-checkpoint=0 --check-requirements=0 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/config1.toml --log-file /tmp/lightning_test/lightning-add-index1.log
Verbose debug logs will be written to /tmp/lightning_test/lightning-add-index1.log

tidb lightning exit successfully
PASS
coverage: 23.2% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 15:02:29 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index --config' /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/config2.toml --log-file '/tmp/lightning_test/lightning-add-index2.log >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_add_index.23170.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/data --sorted-kv-dir /tmp/lightning_test/lightning_add_index.sorted --enable-checkpoint=0 --check-requirements=0 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/config2.toml --log-file /tmp/lightning_test/lightning-add-index2.log
Verbose debug logs will be written to /tmp/lightning_test/lightning-add-index2.log

{
  "version": "v8.2.0-alpha-14-g1679dbca2",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
{
  "version": "v8.2.0-alpha-14-g1679dbca2",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-81-g516977e1f9-dirty","git_hash":"516977e1f99d9df87e142e1ea280baed56a12a70"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }, "log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-81-g516977e1f9-dirty","git_hash":"516977e1f99d9df87e142e1ea280baed56a12a70"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
TiFlash started.
selected test cases: lightning_duplicate_resolution_error
TiFlash started.
selected test cases: lightning_routes
*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error/run.sh... =====*
+ check_cluster_version 5 2 0 'duplicate detection'
++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error/run.sh
+ mydir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error
+ run_sql 'DROP TABLE IF EXISTS dup_resolve.a'
+ run_sql 'DROP TABLE IF EXISTS lightning_task_info.conflict_error_v3'
+ run_lightning --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error/config.toml
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 15:02:46 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error --backend' local --config '/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error/config.toml >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_duplicate_resolution_error.88872.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error/data --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_resolution_error.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error/config.toml
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_routes/run.sh... =====*
+ echo 'testing checksum-error...'
testing checksum-error...
+ export 'GO_FAILPOINTS=github.com/pingcap/tidb/lightning/pkg/importer/checksum-error=1*return()'
+ GO_FAILPOINTS='github.com/pingcap/tidb/lightning/pkg/importer/checksum-error=1*return()'
+ run_sql 'DROP DATABASE IF EXISTS routes_a0;'
+ run_sql 'DROP DATABASE IF EXISTS routes_a1;'
+ run_sql 'DROP DATABASE IF EXISTS routes_b;'
+ run_lightning
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 15:02:46 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_routes  >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_routes.25860.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_routes/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_routes/data --sorted-kv-dir /tmp/lightning_test/lightning_routes.sorted --enable-checkpoint=0 --check-requirements=0
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

tidb lightning encountered error: [Lightning:Restore:ErrFoundDataConflictRecords]found data conflict records in table a, primary key is '3', row data is '(3, 3, "3.csv")'
warning: GOCOVERDIR not set, no coverage data emitted
+ '[' 0 -eq 0 ']'
+ tail -n 10 /tmp/lightning_test/lightning.log
+ grep ERROR
+ tail -n 1
+ grep -Fq '[Lightning:Restore:ErrFoundDataConflictRecords]found data conflict records in table a, primary key is '\''3'\'', row data is '\''(3, 3, \"3.csv\")'\'''
+ check_not_contains 'the whole procedure completed' /tmp/lightning_test/lightning.log
*===== TEST: [lightning_duplicate_resolution_error] success! =====*
tidb lightning exit successfully
PASS
coverage: 23.3% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ echo 'test checksum-error success!'
test checksum-error success!
+ run_sql 'SELECT count(1), sum(x) FROM routes_b.u;'
*************************** 1. row ***************************
count(1): 4
  sum(x): 259
+ check_contains 'count(1): 4'
+ check_contains 'sum(x): 259'
+ run_sql 'SELECT count(1), sum(x) FROM routes_a1.s1;'
*************************** 1. row ***************************
count(1): 1
  sum(x): 1296
+ check_contains 'count(1): 1'
+ check_contains 'sum(x): 1296'
+ run_sql 'SHOW TABLES IN routes_a1;'
*************************** 1. row ***************************
Tables_in_routes_a1: s1
+ check_not_contains 'Tables_in_routes_a1: t2'
*===== TEST: [lightning_routes] success! =====*
Run cases: lightning_duplicate_resolution_error_pk_multiple_files
Starting PD...
Run cases: lightning_routes_panic
Starting PD...
{
  "version": "v8.2.0-alpha-14-g1679dbca2",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
{
  "version": "v8.2.0-alpha-14-g1679dbca2",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-81-g516977e1f9-dirty","git_hash":"516977e1f99d9df87e142e1ea280baed56a12a70"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }, "engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
TiFlash started.
selected test cases: lightning_duplicate_resolution_error_pk_multiple_files
tidb lightning exit successfully
PASS
coverage: 24.3% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
*************************** 1. row ***************************
       Table: multi_indexes
Create Table: CREATE TABLE `multi_indexes` (
  `c1` bigint(20) NOT NULL,
  `c2` varchar(255) NOT NULL,
  `c3` varchar(255) NOT NULL,
  `c4` varchar(255) NOT NULL,
  `c5` varchar(255) NOT NULL,
  `c6` varchar(255) NOT NULL,
  `c7` varchar(255) NOT NULL,
  `c8` varchar(255) NOT NULL,
  `c9` varchar(255) NOT NULL,
  `c10` varchar(255) NOT NULL,
  `c11` varchar(255) NOT NULL,
  PRIMARY KEY (`c1`) /*T![clustered_index] CLUSTERED */,
  KEY `idx_c2` (`c2`) COMMENT 'single column index',
  KEY `idx_c2_c3` (`c2`,`c3`) COMMENT 'multiple column index',
  UNIQUE KEY `uniq_c4` (`c4`) COMMENT 'single column unique key',
  UNIQUE KEY `uniq_c4_c5` (`c4`,`c5`) COMMENT 'multiple column unique key',
  KEY `idx_c6` (`c6`) COMMENT 'single column index with asc order',
  KEY `idx_c7` (`c7`) COMMENT 'single column index with desc order',
  KEY `idx_c6_c7` (`c6`,`c7`) COMMENT 'multiple column index with asc and desc order',
  KEY `idx_c8` (`c8`) COMMENT 'single column index with visible',
  KEY `idx_c9` (`c9`) /*!80000 INVISIBLE */ COMMENT 'single column index with invisible',
  KEY `idx_lower_c10` ((lower(`c10`))) COMMENT 'single column index with function',
  KEY `idx_prefix_c11` (`c11`(3)) COMMENT 'single column index with prefix',
  UNIQUE KEY `c2` (`c2`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 15:03:03 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index --enable-checkpoint=1' --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/config2.toml --log-file '/tmp/lightning_test/lightning-add-index2.log >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_add_index.23306.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/data --sorted-kv-dir /tmp/lightning_test/lightning_add_index.sorted --enable-checkpoint=0 --check-requirements=0 --enable-checkpoint=1 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/config2.toml --log-file /tmp/lightning_test/lightning-add-index2.log
Verbose debug logs will be written to /tmp/lightning_test/lightning-add-index2.log

tidb lightning canceled
PASS
coverage: 25.3% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 15:03:07 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index --enable-checkpoint=1' --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/config2.toml --log-file '/tmp/lightning_test/lightning-add-index2.log >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_add_index.23360.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/data --sorted-kv-dir /tmp/lightning_test/lightning_add_index.sorted --enable-checkpoint=0 --check-requirements=0 --enable-checkpoint=1 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_add_index/config2.toml --log-file /tmp/lightning_test/lightning-add-index2.log
Verbose debug logs will be written to /tmp/lightning_test/lightning-add-index2.log

{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-81-g516977e1f9-dirty","git_hash":"516977e1f99d9df87e142e1ea280baed56a12a70"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }, "log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_pk_multiple_files/run.sh... =====*
+ check_cluster_version 5 2 0 'duplicate detection'
++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_pk_multiple_files/run.sh
+ mydir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_pk_multiple_files
+ run_sql 'DROP TABLE IF EXISTS dup_resolve.a'
+ run_sql 'DROP TABLE IF EXISTS lightning_task_info.conflict_error_v3'
+ run_lightning --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_pk_multiple_files/config.toml
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 15:03:10 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_pk_multiple_files --backend' local --config '/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_pk_multiple_files/config.toml >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_duplicate_resolution_error_pk_multiple_files.90694.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_pk_multiple_files/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_pk_multiple_files/data --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_resolution_error_pk_multiple_files.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_pk_multiple_files/config.toml
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

tidb lightning encountered error: failed to record conflict errors: [Lightning:Restore:ErrFoundDataConflictRecords]found data conflict records in table a, primary key is '1', row data is '(1, 1, "1.csv")'
warning: GOCOVERDIR not set, no coverage data emitted
+ '[' 0 -eq 0 ']'
+ tail -n 10 /tmp/lightning_test/lightning.log
+ grep ERROR
+ tail -n 1
+ grep -Fq '[Lightning:Restore:ErrFoundDataConflictRecords]found data conflict records in table a'
+ check_not_contains 'the whole procedure completed' /tmp/lightning_test/lightning.log
*===== TEST: [lightning_duplicate_resolution_error_pk_multiple_files] success! =====*
Run cases: lightning_duplicate_resolution_error_uk_multiple_files
TiFlash started.
selected test cases: lightning_routes_panic
Starting PD...
*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_routes_panic/run.sh... =====*
+ run_sql 'DROP DATABASE IF EXISTS test1;'
+ run_sql 'DROP DATABASE IF EXISTS test;'
+ run_sql 'CREATE DATABASE test1;'
+ run_sql 'CREATE DATABASE test;'
+ run_sql 'CREATE TABLE test1.dump_test (x real primary key);'
+ run_sql 'CREATE TABLE test.u (x real primary key);'
+ run_lightning
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 15:03:16 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_routes_panic  >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_routes_panic.27769.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_routes_panic/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_routes_panic/data --sorted-kv-dir /tmp/lightning_test/lightning_routes_panic.sorted --enable-checkpoint=0 --check-requirements=0
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

tidb lightning exit successfully
PASS
coverage: 22.7% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ run_sql 'SELECT sum(x) FROM test.u;'
*************************** 1. row ***************************
sum(x): 43
+ check_contains 'sum(x): 43'
*===== TEST: [lightning_routes_panic] success! =====*
{
  "version": "v8.2.0-alpha-14-g1679dbca2",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
Run cases: lightning_row-format-v2
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
Starting PD...
{
  "version": "v8.2.0-alpha-14-g1679dbca2",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-81-g516977e1f9-dirty","git_hash":"516977e1f99d9df87e142e1ea280baed56a12a70"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }, "log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
TiFlash started.
selected test cases: lightning_duplicate_resolution_error_uk_multiple_files
*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files/run.sh... =====*
+ check_cluster_version 5 2 0 'duplicate detection'
++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files/run.sh
+ mydir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files
+ run_sql 'DROP TABLE IF EXISTS dup_resolve.a'
+ run_sql 'DROP TABLE IF EXISTS lightning_task_info.conflict_error_v3'
+ run_lightning --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files/config.toml
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 15:03:34 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files --backend' local --config '/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files/config.toml >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_duplicate_resolution_error_uk_multiple_files.92519.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files/data --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_resolution_error_uk_multiple_files.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files/config.toml
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

tidb lightning encountered error: [Lightning:Restore:ErrFoundIndexConflictRecords]found index conflict records in table a, index name is 'a.key_b', unique key is '[101]', primary key is '1'
warning: GOCOVERDIR not set, no coverage data emitted
+ '[' 0 -eq 0 ']'
+ tail -n 10 /tmp/lightning_test/lightning.log
+ grep ERROR
+ tail -n 1
+ grep -Fq '[Lightning:Restore:ErrFoundIndexConflictRecords]found index conflict records in table a, index name is '\''a.key_b'\'', unique key is '\''[101]'\'', primary key is '\''1'\'''
+ check_not_contains 'the whole procedure completed' /tmp/lightning_test/lightning.log
*===== TEST: [lightning_duplicate_resolution_error_uk_multiple_files] success! =====*
Run cases: lightning_duplicate_resolution_error_uk_multiple_files_multicol_index
Starting PD...
tidb lightning exit successfully
PASS
coverage: 13.7% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ '[' 130 == 130 ']'
+ '[' 12548524131860600223 == 12548524131860600223 ']'
+ '[' 30 == 30 ']'
+ '[' 3322292718850045174 == 3322292718850045174 ']'
*===== TEST: [lightning_add_index] success! =====*
Run cases: lightning_alter_random
Starting PD...
{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-81-g516977e1f9-dirty","git_hash":"516977e1f99d9df87e142e1ea280baed56a12a70"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
{
  "version": "v8.2.0-alpha-14-g1679dbca2",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
TiFlash started.
selected test cases: lightning_row-format-v2
{
  "version": "v8.2.0-alpha-14-g1679dbca2",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_row-format-v2/run.sh... =====*
+ run_sql 'show variables like "%tidb_row_format_version%";'
*************************** 1. row ***************************
Variable_name: tidb_row_format_version
        Value: 2
++ grep 'Value: [0-9]' /tmp/lightning_test/sql_res.lightning_row-format-v2.txt
++ awk '{print $2}'
+ row_format=2
+ '[' 2 -ne 2 ']'
+ run_sql 'DROP DATABASE IF EXISTS rowformatv2;'
+ run_lightning
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 15:03:46 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_row-format-v2  >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_row-format-v2.29631.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_row-format-v2/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_row-format-v2/data --sorted-kv-dir /tmp/lightning_test/lightning_row-format-v2.sorted --enable-checkpoint=0 --check-requirements=0
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
tidb lightning exit successfully
PASS
coverage: 22.8% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ run_sql 'SELECT count(1) FROM rowformatv2.t1;'
*************************** 1. row ***************************
count(1): 50
+ check_contains 'count(1): 50'
+ run_sql 'SELECT DISTINCT col14 FROM rowformatv2.t1;'
*************************** 1. row ***************************
col14: 39
*************************** 2. row ***************************
col14: NULL
+ check_contains 'col14: NULL'
+ check_contains 'col14: 39'
+ '[' 2 -ne 2 ']'
*===== TEST: [lightning_row-format-v2] success! =====*
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
Run cases: lightning_s3
Starting PD...
{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-81-g516977e1f9-dirty","git_hash":"516977e1f99d9df87e142e1ea280baed56a12a70"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }, "log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
{
  "version": "v8.2.0-alpha-14-g1679dbca2",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-81-g516977e1f9-dirty","git_hash":"516977e1f99d9df87e142e1ea280baed56a12a70"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
TiFlash started.
selected test cases: lightning_duplicate_resolution_error_uk_multiple_files_multicol_index
TiFlash started.
selected test cases: lightning_alter_random
*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files_multicol_index/run.sh... =====*
+ check_cluster_version 5 2 0 'duplicate detection'
++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files_multicol_index/run.sh
+ mydir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files_multicol_index
+ run_sql 'DROP TABLE IF EXISTS dup_resolve.a'
+ run_sql 'DROP TABLE IF EXISTS lightning_task_info.conflict_error_v3'
+ run_lightning --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files_multicol_index/config.toml
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 15:03:59 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files_multicol_index --backend' local --config '/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files_multicol_index/config.toml >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_duplicate_resolution_error_uk_multiple_files_multicol_index.94367.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files_multicol_index/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files_multicol_index/data --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_resolution_error_uk_multiple_files_multicol_index.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_error_uk_multiple_files_multicol_index/config.toml
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

tidb lightning encountered error: [Lightning:Restore:ErrFoundIndexConflictRecords]found index conflict records in table a, index name is 'a.key_bd', unique key is '[101 9]', primary key is '1'
warning: GOCOVERDIR not set, no coverage data emitted
+ '[' 0 -eq 0 ']'
+ tail -n 10 /tmp/lightning_test/lightning.log
+ grep ERROR
+ tail -n 1
+ grep -Fq '[Lightning:Restore:ErrFoundIndexConflictRecords]found index conflict records in table a, index name is '\''a.key_bd'\'', unique key is '\''[101 9]'\'', primary key is '\''1'\'''
+ check_not_contains 'the whole procedure completed' /tmp/lightning_test/lightning.log
*===== TEST: [lightning_duplicate_resolution_error_uk_multiple_files_multicol_index] success! =====*
*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_alter_random/run.sh... =====*
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 15:04:00 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_alter_random --backend' 'tidb >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_alter_random.25230.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_alter_random/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_alter_random/data --sorted-kv-dir /tmp/lightning_test/lightning_alter_random.sorted --enable-checkpoint=0 --check-requirements=0 --backend tidb
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

tidb lightning exit successfully
PASS
coverage: 14.7% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
*************************** 1. row ***************************
count(*): 3
*************************** 1. row ***************************
inc: 1
*************************** 2. row ***************************
inc: 3
*************************** 3. row ***************************
inc: 2
*************************** 1. row ***************************
inc: 30002
*************************** 2. row ***************************
inc: 1
*************************** 3. row ***************************
inc: 3
*************************** 4. row ***************************
inc: 2
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 15:04:01 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_alter_random --backend' 'local >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_alter_random.25294.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_alter_random/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_alter_random/data --sorted-kv-dir /tmp/lightning_test/lightning_alter_random.sorted --enable-checkpoint=0 --check-requirements=0 --backend local
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

Run cases: lightning_duplicate_resolution_incremental
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
tidb lightning exit successfully
PASS
coverage: 22.9% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
*************************** 1. row ***************************
count(*): 3
*************************** 1. row ***************************
inc: 1
*************************** 2. row ***************************
inc: 3
*************************** 3. row ***************************
inc: 2
*************************** 1. row ***************************
inc: 4
*************************** 2. row ***************************
inc: 1
*************************** 3. row ***************************
inc: 3
*************************** 4. row ***************************
inc: 2
*===== TEST: [lightning_alter_random] success! =====*
Starting PD...
Run cases: lightning_auto_columns
Starting PD...
{
  "version": "v8.2.0-alpha-14-g1679dbca2",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-81-g516977e1f9-dirty","git_hash":"516977e1f99d9df87e142e1ea280baed56a12a70"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
{
  "version": "v8.2.0-alpha-14-g1679dbca2",
  "build_time": "",
  "hash": "",
  "branch": ""
}
Starting TiKV(1)...
Starting TiKV(2)...
Starting TiKV(3)...
Waiting initializing TiKV...
TiFlash started.
selected test cases: lightning_s3
  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_s3/run.sh... =====*
+ DB=s3_test
+ TABLE=tbl
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_s3/run.sh
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_s3
++ pwd
+ CUR=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_s3
+ check_cluster_version 4 0 0 'local backend'
+ set -euE
+ [[ -z /tmp/lightning_test ]]
+ DBPATH=/tmp/lightning_test/s3.mydump
+ export MINIO_ACCESS_KEY=s3accesskey
+ MINIO_ACCESS_KEY=s3accesskey
+ export MINIO_SECRET_KEY=s3secretkey
+ MINIO_SECRET_KEY=s3secretkey
+ export MINIO_BROWSER=off
+ MINIO_BROWSER=off
+ export S3_ENDPOINT=127.0.0.1:9900
+ S3_ENDPOINT=127.0.0.1:9900
+ rm -rf /tmp/lightning_test/s3_test
+ mkdir -p /tmp/lightning_test/s3_test
+ i=0
+ curl -o /dev/null -v -s http://127.0.0.1:9900/
+ minio server --address 127.0.0.1:9900 /tmp/lightning_test/s3.mydump
* About to connect() to 127.0.0.1 port 9900 (#0)
*   Trying 127.0.0.1...
* Connection refused
* Failed connect to 127.0.0.1:9900; Connection refused
* Closing connection 0
+ i=1
+ '[' 1 -gt 30 ']'
+ sleep 2

 You are running an older version of MinIO released 3 years ago 
 Update: Run `mc admin update` 


Attempting encryption of all config, IAM users and policies on MinIO backend
Endpoint: http://127.0.0.1:9900 

Object API (Amazon S3 compatible):
   Go:         https://docs.min.io/docs/golang-client-quickstart-guide
   Java:       https://docs.min.io/docs/java-client-quickstart-guide
   Python:     https://docs.min.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
   .NET:       https://docs.min.io/docs/dotnet-client-quickstart-guide
+ curl -o /dev/null -v -s http://127.0.0.1:9900/
* About to connect() to 127.0.0.1 port 9900 (#0)
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 9900 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:9900
> Accept: */*
> 
< HTTP/1.1 403 Forbidden
< Accept-Ranges: bytes
< Content-Length: 226
< Content-Security-Policy: block-all-mixed-content
< Content-Type: application/xml
< Server: MinIO
< Vary: Origin
< X-Amz-Request-Id: 17CCD45D1CC610D0
< X-Xss-Protection: 1; mode=block
< Date: Mon, 06 May 2024 07:04:14 GMT
< 
{ [data not shown]
* Connection #0 to host 127.0.0.1 left intact
+ BUCKET=test-bucket
+ DATA_PATH=/tmp/lightning_test/s3.mydump/test-bucket
+ mkdir -p /tmp/lightning_test/s3.mydump/test-bucket
+ echo 'CREATE DATABASE s3_test;'
+ echo 'CREATE TABLE t(i INT, s varchar(32));'
+ echo 'INSERT INTO tbl (i, s) VALUES (1, "1"),(2, "test2"), (3, "qqqtest");'
+ cat
+ final_ret_code=0
+ test_import_non_existing_path
+ rm -f /tmp/lightning_test/lightning.log
+ local 'SOURCE_DIR=s3://test-bucket/not-exist-path?endpoint=http%3A//127.0.0.1%3A9900&access_key=s3accesskey&secret_access_key=s3secretkey&force_path_style=true'
+ run_lightning -d 's3://test-bucket/not-exist-path?endpoint=http%3A//127.0.0.1%3A9900&access_key=s3accesskey&secret_access_key=s3secretkey&force_path_style=true' --backend local
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

+ grep -Eq 'data-source-dir .* doesn'\''t exist or contains no files' /tmp/lightning_test/lightning.log
+ return 0
+ test_import_empty_dir
+ rm -f /tmp/lightning_test/lightning.log
+ local emptyPath=empty-bucket/empty-path
+ mkdir -p /tmp/lightning_test/s3.mydump/empty-bucket/empty-path
+ local 'SOURCE_DIR=s3://empty-bucket/empty-path/not-exist-path?endpoint=http%3A//127.0.0.1%3A9900&access_key=s3accesskey&secret_access_key=s3secretkey&force_path_style=true'
+ run_lightning -d 's3://empty-bucket/empty-path/not-exist-path?endpoint=http%3A//127.0.0.1%3A9900&access_key=s3accesskey&secret_access_key=s3secretkey&force_path_style=true' --backend local
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

+ grep -Eq 'data-source-dir .* doesn'\''t exist or contains no files' /tmp/lightning_test/lightning.log
+ return 0
+ test_normal_import
+ rm -f /tmp/lightning_test/lightning.log
+ cleanup_db_and_table
+ run_sql 'DROP DATABASE IF EXISTS s3_test;'
+ run_sql 'DROP TABLE IF EXISTS s3_test.tbl;'
+ local 'SOURCE_DIR=s3://test-bucket/?endpoint=http%3A//127.0.0.1%3A9900&access_key=s3accesskey&secret_access_key=s3secretkey&force_path_style=true'
+ run_lightning -d 's3://test-bucket/?endpoint=http%3A//127.0.0.1%3A9900&access_key=s3accesskey&secret_access_key=s3secretkey&force_path_style=true' --backend local
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

  "is_initialized": true,
Starting TiDB...
Verifying TiDB is started...
tidb lightning exit successfully
PASS
coverage: 24.9% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ run_sql 'SELECT count(*), sum(i) FROM `s3_test`.tbl'
*************************** 1. row ***************************
count(*): 7
  sum(i): 413
+ check_contains 'count(*): 7'
+ check_contains 'sum(i): 413'
+ return 0
+ test_import_with_checkpoint
+ rm -f /tmp/lightning_test/lightning.log
+ cleanup_db_and_table
+ run_sql 'DROP DATABASE IF EXISTS s3_test;'
+ run_sql 'DROP TABLE IF EXISTS s3_test.tbl;'
+ local 'SOURCE_DIR=s3://test-bucket/?endpoint=http%3A//127.0.0.1%3A9900&access_key=s3accesskey&secret_access_key=s3secretkey&force_path_style=true'
+ run_lightning -d 's3://test-bucket/?endpoint=http%3A//127.0.0.1%3A9900&access_key=s3accesskey&secret_access_key=s3secretkey&force_path_style=true' --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_s3/config_s3_checkpoint.toml
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

tidb lightning exit successfully
PASS
coverage: 25.1% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ run_sql 'SELECT count(*), sum(i) FROM `s3_test`.tbl'
*************************** 1. row ***************************
count(*): 7
  sum(i): 413
+ check_contains 'count(*): 7'
+ check_contains 'sum(i): 413'
+ test_import_using_manual_path_config
+ rm -f /tmp/lightning_test/lightning.log
+ cleanup_db_and_table
+ run_sql 'DROP DATABASE IF EXISTS s3_test;'
+ run_sql 'DROP TABLE IF EXISTS s3_test.tbl;'
+ local bucket_02=test-bucket-02
+ local sub_path_02=to-be-imported
+ local base_path_02=/tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported
+ rm -rf /tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported
+ mkdir -p /tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported
+ touch /tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported/s3_test-schema-create.sql
+ cp /tmp/lightning_test/s3.mydump/test-bucket/s3_test.tbl-schema.sql /tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported/
+ local sql_data_path_02=/tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported/data-sql
+ local csv_data_path_02=/tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported/data-csv
+ local parquet_data_path_02=/tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported/data-parquet
+ mkdir -p /tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported/data-sql/empty-dir
+ cp /tmp/lightning_test/s3.mydump/test-bucket/s3_test.tbl.sql /tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported/data-sql/
+ touch /tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported/data-sql/dummy.sql
+ mkdir -p /tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported/data-csv/empty-dir
+ cp /tmp/lightning_test/s3.mydump/test-bucket/s3_test.tbl.0.csv /tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported/data-csv/
+ touch /tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported/data-csv/dummy.csv
+ mkdir -p /tmp/lightning_test/s3.mydump/test-bucket-02/to-be-imported/data-parquet/empty-dir
+ local 'SOURCE_DIR=s3://test-bucket-02/to-be-imported?endpoint=http%3A//127.0.0.1%3A9900&access_key=s3accesskey&secret_access_key=s3secretkey&force_path_style=true'
+ run_lightning -d 's3://test-bucket-02/to-be-imported?endpoint=http%3A//127.0.0.1%3A9900&access_key=s3accesskey&secret_access_key=s3secretkey&force_path_style=true' --backend local --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_s3/config_manual_files.toml
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-81-g516977e1f9-dirty","git_hash":"516977e1f99d9df87e142e1ea280baed56a12a70"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
tidb lightning exit successfully
PASS
coverage: 24.9% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ run_sql 'SELECT count(*), sum(i) FROM `s3_test`.tbl'
*************************** 1. row ***************************
count(*): 7
  sum(i): 413
+ check_contains 'count(*): 7'
+ check_contains 'sum(i): 413'
+ return 0
+ exit 0
*===== TEST: [lightning_s3] success! =====*
TiFlash started.
selected test cases: lightning_duplicate_resolution_incremental
[Pipeline] }
Cache not saved (ws/jenkins-pingcap-tidb-merged_integration_lightning_test-533/lightning-tests already exists)
[Pipeline] // cache
{"connections":0,"version":"8.0.11-TiDB-v8.2.0-alpha-81-g516977e1f9-dirty","git_hash":"516977e1f99d9df87e142e1ea280baed56a12a70"}Starting TiFlash...
TiFlash seems doesn't started, retrying...
Logging debug to /tmp/lightning_test/tiflash-stdout.log
Logging errors to /tmp/lightning_test/tiflash-stderr.log
arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8e50de84e6d6ecdcc108990217b70b6bb3f50271"] }, "engine-version": MatchedArg { occurs: 1, indices: [10], vals: ["v8.2.0-alpha-17-g8e50de84e"] }, "log-file": MatchedArg { occurs: 1, indices: [2], vals: ["/tmp/lightning_test/tiflash-proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [6], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [4], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/lightning_test/tiflash-learner.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [14], vals: ["127.0.0.1:3930"] }}, subcommand: None, usage: Some("USAGE:\n    TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash <engine-git-hash> --engine-label <engine-label> --engine-version <engine-version>") }
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental/run.sh... =====*
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental/run.sh
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental
++ pwd
+ CUR=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental
+ check_cluster_version 5 2 0 'duplicate detection'
+ LOG_FILE1=/tmp/lightning_test/lightning-duplicate-resolution1.log
+ LOG_FILE2=/tmp/lightning_test/lightning-duplicate-resolution2.log
+ export 'GO_FAILPOINTS=github.com/pingcap/tidb/lightning/pkg/importer/SlowDownCheckDupe=return(10)'
+ GO_FAILPOINTS='github.com/pingcap/tidb/lightning/pkg/importer/SlowDownCheckDupe=return(10)'
+ counter=0
+ '[' 0 -lt 10 ']'
+ grep -Fq 'start to sleep several seconds before checking other dupe' /tmp/lightning_test/lightning-duplicate-resolution1.log
+ run_lightning --backend local --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_resolution_incremental.sorted1 --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-duplicate-resolution1.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental/config1.toml
grep: /tmp/lightning_test/lightning-duplicate-resolution1.log: No such file or directory
+ (( counter += 1 ))
+ echo 'waiting for lightning 1 starts'
waiting for lightning 1 starts
+ sleep 1
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 15:04:23 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental --backend' local --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_resolution_incremental.sorted1 --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-duplicate-resolution1.log --config '/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental/config1.toml >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_duplicate_resolution_incremental.96217.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental/data --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_resolution_incremental.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_resolution_incremental.sorted1 --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-duplicate-resolution1.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental/config1.toml
Verbose debug logs will be written to /tmp/lightning_test/lightning-duplicate-resolution1.log

TiFlash started.
selected test cases: lightning_auto_columns
+ '[' 1 -lt 10 ']'
+ grep -Fq 'start to sleep several seconds before checking other dupe' /tmp/lightning_test/lightning-duplicate-resolution1.log
+ (( counter += 1 ))
+ echo 'waiting for lightning 1 starts'
waiting for lightning 1 starts
+ sleep 1
+ '[' 2 -lt 10 ']'
+ grep -Fq 'start to sleep several seconds before checking other dupe' /tmp/lightning_test/lightning-duplicate-resolution1.log
+ echo 'lightning 1 already starts waiting for dupe'
lightning 1 already starts waiting for dupe
+ break
+ '[' 2 -ge 10 ']'
+ wait
+ run_lightning --backend local --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_resolution_incremental.sorted2 --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-duplicate-resolution2.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental/config2.toml
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 15:04:25 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental --backend' local --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_resolution_incremental.sorted2 --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-duplicate-resolution2.log --config '/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental/config2.toml >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_duplicate_resolution_incremental.96269.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental/data --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_resolution_incremental.sorted --enable-checkpoint=0 --check-requirements=0 --backend local --sorted-kv-dir /tmp/lightning_test/lightning_duplicate_resolution_incremental.sorted2 --enable-checkpoint=1 --log-file /tmp/lightning_test/lightning-duplicate-resolution2.log --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_duplicate_resolution_incremental/config2.toml
Verbose debug logs will be written to /tmp/lightning_test/lightning-duplicate-resolution2.log

*===== Running test /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_auto_columns/run.sh... =====*
+ run_sql 'DROP DATABASE IF EXISTS lightning_auto_cols;'
+ run_lightning
+++ dirname /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils/run_lightning
++ cd /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/../../tests/_utils
++ pwd
+ cur_dir=/home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/tests/_utils
++ date
+ echo '[Mon May  6 15:04:25 CST 2024] <<<<<< RUNNING TEST FOR: /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_auto_columns  >>>>>>'
+ tidb-lightning.test -test.coverprofile=/tmp/group_cover/cov.lightning_auto_columns.27138.out DEVEL --ca /tmp/lightning_test/certs/ca.pem --cert /tmp/lightning_test/certs/lightning.pem --key /tmp/lightning_test/certs/lightning.key --log-file /tmp/lightning_test/lightning.log --tidb-port 4000 --pd-urls 127.0.0.1:2379 --config /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_auto_columns/config.toml -d /home/jenkins/agent/workspace/pingcap/tidb/merged_integration_lightning_test/tidb/lightning/tests/lightning_auto_columns/data --sorted-kv-dir /tmp/lightning_test/lightning_auto_columns.sorted --enable-checkpoint=0 --check-requirements=0
Verbose debug logs will be written to /tmp/lightning_test/lightning.log

tidb lightning exit successfully
PASS
coverage: 23.5% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ run_sql 'SELECT CONCAT_WS('\'':'\'', id, c) AS row_data FROM lightning_auto_cols.t_auto_incr;'
*************************** 1. row ***************************
row_data: 1:normal_pk_01
*************************** 2. row ***************************
row_data: 2:null_pk_02
*************************** 3. row ***************************
row_data: 3:null_pk_03
*************************** 4. row ***************************
row_data: 4:normal_pk_04
+ check_contains 'row_data: 1:normal_pk_01'
+ check_contains 'row_data: 2:null_pk_02'
+ check_contains 'row_data: 3:null_pk_03'
+ check_contains 'row_data: 4:normal_pk_04'
+ run_sql 'SELECT COUNT(*) AS row_count FROM lightning_auto_cols.t_auto_incr;'
*************************** 1. row ***************************
row_count: 4
+ check_contains 'row_count: 4'
+ run_sql 'SELECT CONCAT_WS('\'':'\'', id, c) AS row_data FROM lightning_auto_cols.t_auto_random;'
*************************** 1. row ***************************
row_data: 1:normal_pk_01
*************************** 2. row ***************************
row_data: 4:normal_pk_04
*************************** 3. row ***************************
row_data: 1152921504606846978:null_pk_02
*************************** 4. row ***************************
row_data: 1152921504606846979:null_pk_03
+ check_contains 'row_data: 1:normal_pk_01'
+ check_contains :null_pk_02
+ check_not_contains 'row_data: 0:null_pk_02'
+ check_contains :null_pk_03
+ check_not_contains 'row_data: 0:null_pk_03'
+ check_contains 'row_data: 4:normal_pk_04'
+ run_sql 'SELECT COUNT(*) AS row_count FROM lightning_auto_cols.t_auto_random;'
*************************** 1. row ***************************
row_count: 4
+ check_contains 'row_count: 4'
*===== TEST: [lightning_auto_columns] success! =====*
[Pipeline] }
Cache not saved (ws/jenkins-pingcap-tidb-merged_integration_lightning_test-533/lightning-tests already exists)
[Pipeline] // cache
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

Import Data Error Summary: 
+---+---------------------+-------------+--------------------------------------------+
| # | ERROR TYPE          | ERROR COUNT | ERROR DATA TABLE                           |
+---+---------------------+-------------+--------------------------------------------+
|[31m 1 [0m|[31m Unique Key Conflict [0m|[31m          10 [0m|[31m `lightning_task_info_dupe_resolve_incremen [0m|
|[31m   [0m|[31m                     [0m|[31m             [0m|[31m tal`.`conflict_error_v3`                   [0m|
+---+---------------------+-------------+--------------------------------------------+

tidb lightning exit successfully
PASS
coverage: 27.2% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
tidb lightning exit successfully
PASS
coverage: 28.3% of statements in github.com/pingcap/tidb/lightning/..., github.com/pingcap/tidb/pkg/lightning/...
+ export GO_FAILPOINTS=
+ GO_FAILPOINTS=
+ run_sql 'admin check table dup_resolve_detect.ta'
+ run_sql 'select count(*), sum(id) from dup_resolve_detect.ta where id < 100'
*************************** 1. row ***************************
count(*): 15
 sum(id): 120
+ check_contains 'count(*): 15'
+ check_contains 'sum(id): 120'
+ run_sql 'select count(*), sum(id) from dup_resolve_detect.ta where id > 100'
*************************** 1. row ***************************
count(*): 16
 sum(id): 1896
+ check_contains 'count(*): 16'
+ check_contains 'sum(id): 1896'
*===== TEST: [lightning_duplicate_resolution_incremental] success! =====*
[Pipeline] }
Cache not saved (ws/jenkins-pingcap-tidb-merged_integration_lightning_test-533/lightning-tests already exists)
[Pipeline] // cache
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Finished: FAILURE