Skipping 798 KB..
Full LogTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTest
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
[Sat May 4 17:21:41 CST 2024] <<<<<< start DM-007 >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl1_1/conf/double-source-pessimistic.yaml --remove-meta"
dmctl test cmd: "operate-source show"
got=1 expected=1
dmctl test cmd: "query-status -s mysql-replica-01"
rpc addr 127.0.0.1:8264 is alive
start DM task
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/lightning_load_task/conf/dm-task.yaml --remove-meta"
dmctl test cmd: "purge-relay --filename dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000027 -s mysql-replica-01"
earliest_relay_log1: dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001 earliest_relay_log2: dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001
purge relay log failed 7-th time, retry later
got=2 expected=2
check diff successfully
>>>>>>>>>>>>>>>>>>>>>>>>>>test test_relay_operations passed
1 dm-master alive
2 dm-worker alive
0 dm-syncer alive
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/full_mode/source2.yaml"
check log contain failed 23-th time, retry later
dmctl test cmd: "purge-relay --filename dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000028 -s mysql-replica-01"
earliest_relay_log1: dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001 earliest_relay_log2: dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001
purge relay log failed 8-th time, retry later
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/validator_basic/conf/dm-task-standalone-no-validator.yaml --remove-meta"
dmctl test cmd: "query-status test"
got=2 expected=2
check diff failed 1-th time, retry later
check diff failed 3-th time, retry later
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/lightning_load_task/conf/dm-task2.yaml --remove-meta"
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/full_mode/conf/dm-task.yaml --remove-meta"
wait process dm-master.test exit...
dmctl test cmd: "config source mysql-replica-01"
got=1 expected=1
got=1 expected=1
dmctl test cmd: "purge-relay --filename dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000029 -s mysql-replica-01"
earliest_relay_log1: dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001 earliest_relay_log2: dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001
purge relay log failed 9-th time, retry later
wait process dm-master.test exit...
process dm-master.test already exit
dmctl test cmd: "query-status load_task1"
got=1 expected=1
got=1 expected=1
dmctl test cmd: "query-status load_task2"
got=1 expected=1
got=1 expected=1
test worker restart
check diff successfully
check diff failed 4-th time, retry later
after restart dm-worker, task should resume automatically
HTTP 127.0.0.1:8261/apis/v1alpha1/status/test is alive
dmctl test cmd: "purge-relay --filename dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000030 -s mysql-replica-02"
check log contain failed 24-th time, retry later
wait process dm-worker.test exit...
check diff failed 1-th time, retry later
earliest_relay_log1: dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001 earliest_relay_log2: dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001
purge relay log failed 1-th time, retry later
check diff successfully
dmctl test cmd: "stop-task test"
[Sat May 4 17:21:44 CST 2024] <<<<<< finish DM-007 >>>>>>
wait process dm-worker1 exit...
dmctl test cmd: "query-status test"
1 dm-master alive
2 dm-worker alive
0 dm-syncer alive
dmctl test cmd: "query-status test"
got=1 expected=1
check diff successfully
dmctl test cmd: "validation status test"
got=1 expected=1
dmctl test cmd: "validation start test"
wait process dm-worker.test exit...
dmctl test cmd: "purge-relay --filename dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000032 -s mysql-replica-02"
earliest_relay_log1: dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001 earliest_relay_log2: dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001
purge relay log failed 2-th time, retry later
dmctl test cmd: "validation status test"
got=1 expected=1
[Sat May 4 17:21:46 CST 2024] <<<<<< start DM-008 >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl1_1/conf/double-source-pessimistic.yaml --remove-meta"
wait process dm-master.test exit...
wait process dm-worker1 exit...
process dm-worker1 already exit
dmctl test cmd: "list-member -w -n worker3"
got=1 expected=1
got=1 expected=1
dmctl test cmd: "list-member -w -n worker1"
got=1 expected=1
dmctl test cmd: "query-status load_task1"
check log contain failed 25-th time, retry later
got=1 expected=1
got=1 expected=1
[Sat May 4 17:21:46 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/lightning_load_task/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 17:21:46 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/new_relay/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
check diff successfully
check diff failed 5-th time, retry later
wait process dm-master.test exit...
process dm-master.test already exit
dmctl test cmd: "purge-relay --filename dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000033 -s mysql-replica-02"
earliest_relay_log1: dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001 earliest_relay_log2: dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001
purge relay log failed 3-th time, retry later
dmctl test cmd: "query-status test"
got=2 expected=2
check diff failed 1-th time, retry later
rpc addr 127.0.0.1:8262 is alive
dmctl test cmd: "list-member -w -n worker3"
dmctl test cmd: "validation status test"
got=0 expected=1
command: validation status test "processedRowsStatus": "insert\/update\/delete: 2\/1\/1" count: 0 != expected: 1, failed the 0-th time, will retry again
got=1 expected=1
dmctl test cmd: "list-member -w -n worker1"
got=1 expected=1
got=1 expected=1
dmctl test cmd: "query-status load_task1"
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 17:21:47 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/new_relay/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
dmctl test cmd: "purge-relay --filename dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000034 -s mysql-replica-02"
got=1 expected=1
got=1 expected=1
dmctl test cmd: "query-status load_task2"
got=1 expected=1
got=1 expected=1
test_transfer_two_sources
earliest_relay_log1: dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001 earliest_relay_log2: dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001
purge relay log failed 4-th time, retry later
check log contain failed 26-th time, retry later
wait process dm-worker.test exit...
check diff failed 6-th time, retry later
rpc addr 127.0.0.1:8262 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/new_relay/source1.yaml"
wait process dm-worker2 exit...
dmctl test cmd: "start-relay -s mysql-replica-01 worker1"
dmctl test cmd: "stop-task test"
dmctl test cmd: "operate-source stop /tmp/dm_test/case_sensitive/source1.yaml"
dmctl test cmd: "operate-source stop /tmp/dm_test/case_sensitive/source2.yaml"
1 dm-master alive
dmctl test cmd: "purge-relay --filename dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000035 -s mysql-replica-02"
wait process dm-worker.test exit...
2 dm-worker alive
0 dm-syncer alive
earliest_relay_log1: dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001 earliest_relay_log2: dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001
purge relay log failed 5-th time, retry later
got=1 expected=1
got=0 expected=1
command: validation status test pendingRowsStatus": "insert\/update\/delete: 0\/0\/0 count: 0 != expected: 1, failed the 1-th time, will retry again
check diff successfully
dmctl test cmd: "stop-task test"
[Sat May 4 17:21:49 CST 2024] <<<<<< finish DM-008 >>>>>>
got=2 expected=2
kill dm-worker1
check log contain failed 27-th time, retry later
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 17:21:50 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/full_mode/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
wait process dm-worker2 exit...
process dm-worker2 already exit
dmctl test cmd: "list-member -w -n worker3"
got=1 expected=1
got=1 expected=1
dmctl test cmd: "query-status load_task2"
got=1 expected=1
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/lightning_load_task/conf/dm-task3.yaml --remove-meta"
wait process dm-master.test exit...
dmctl test cmd: "purge-relay --filename dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000036 -s mysql-replica-02"
earliest_relay_log1: dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001 earliest_relay_log2: dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001
purge relay log failed 6-th time, retry later
check diff failed 7-th time, retry later
[Sat May 4 17:21:50 CST 2024] <<<<<< start DM-009 >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl1_1/conf/double-source-pessimistic.yaml --remove-meta"
wait process dm-master.test exit...
process dm-master.test already exit
wait process dm-worker1 exit...
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 17:21:51 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/full_mode/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
got=2 expected=2
dmctl test cmd: "query-status load_task3"
got=1 expected=1
[Sat May 4 17:21:51 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/lightning_load_task/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
dmctl test cmd: "purge-relay --filename dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000037 -s mysql-replica-02"
earliest_relay_log1: dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001 earliest_relay_log2: dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001
purge relay log failed 7-th time, retry later
got=1 expected=1
got=1 expected=1
got=1 expected=1
check diff successfully
--> start validator on the fly, validate from current syncer progress(gtid=false, relay=true)
check log contain failed 28-th time, retry later
wait process dm-worker.test exit...
dmctl test cmd: "query-status test"
got=2 expected=2
wait process dm-worker1 exit...
process dm-worker1 already exit
[Sat May 4 17:21:52 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/new_relay/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
check diff failed 1-th time, retry later
rpc addr 127.0.0.1:8262 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/full_mode/source1.yaml"
[Sat May 4 17:21:52 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/full_mode/conf/dm-worker2.toml >>>>>>
check diff failed 8-th time, retry later
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "list-member -w -n worker2"
dmctl test cmd: "purge-relay --filename dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000038 -s mysql-replica-02"
earliest_relay_log1: dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001 earliest_relay_log2: dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001
purge relay log failed 8-th time, retry later
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
got=1 expected=1
1 dm-master alive
2 dm-worker alive
0 dm-syncer alive
rpc addr 127.0.0.1:8262 is alive
dmctl test cmd: "query-status -s mysql-replica-01"
got=1 expected=1
>>>>>>>>>>>>>>>>>>>>>>>>>>test test_cant_dail_upstream passed
1 dm-master alive
1 dm-worker alive
0 dm-syncer alive
wait process dm-worker.test exit...
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/full_mode/source2.yaml"
wait process dm-worker1 exit...
wait process dm-master.test exit...
dmctl test cmd: "purge-relay --filename dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000039 -s mysql-replica-02"
earliest_relay_log1: dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001 earliest_relay_log2: dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001
purge relay log failed 9-th time, retry later
check log contain failed 29-th time, retry later
check diff successfully
dmctl test cmd: "stop-task test"
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 17:21:54 CST 2024] <<<<<< test case case_sensitive success! >>>>>>
start running case: [checkpoint_transaction] script: [/home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/checkpoint_transaction/run.sh]
Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/checkpoint_transaction/run.sh...
Verbose mode = false
0 dm-master alive
0 dm-worker alive
0 dm-syncer alive
process dm-master.test already exit
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 17:21:54 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/checkpoint_transaction/conf/dm-master.toml >>>>>>
wait process dm-master.test exit...
wait process dm-master.test exit...
process dm-master.test already exit
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
check diff successfully
/home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/only_dml/run.sh: line 44: 5667 Terminated insert_data
1 dm-master alive
2 dm-worker alive
0 dm-syncer alive
[Sat May 4 17:21:54 CST 2024] <<<<<< finish DM-009 >>>>>>
check diff failed 9-th time, retry later
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/full_mode/conf/dm-task-2.yaml --remove-meta"
wait process dm-worker1 exit...
process dm-worker1 already exit
dmctl test cmd: "list-member -w -n worker2"
got=1 expected=1
got=1 expected=1
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/lightning_load_task/conf/dm-task4.yaml --remove-meta"
wait process dm-master.test exit...
process dm-master.test already exit
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 17:21:55 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/checkpoint_transaction/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
[Sat May 4 17:21:55 CST 2024] <<<<<< start DM-010 >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl1_1/conf/single-source-pessimistic.yaml --remove-meta"
dmctl test cmd: "query-status test"
wait process dm-worker.test exit...
wait process dm-master.test exit...
got=1 expected=1
1 dm-master alive
2 dm-worker alive
0 dm-syncer alive
got=2 expected=2
dmctl test cmd: "query-status load_task4"
got=1 expected=1
[Sat May 4 17:21:56 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/lightning_load_task/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
wait process dm-worker.test exit...
check log contain failed 30-th time, retry later
check diff failed 10-th time, retry later
rpc addr 127.0.0.1:8262 is alive
dmctl test cmd: "operate-source create /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/checkpoint_transaction/conf/source1.yaml"
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/checkpoint_transaction/conf/dm-task.yaml "
wait process dm-worker.test exit...
wait process dm-master.test exit...
process dm-master.test already exit
dmctl test cmd: "query-status test"
got=1 expected=1
check diff failed 1-th time, retry later
wait process dm-master.test exit...
rpc addr 127.0.0.1:8262 is alive
dmctl test cmd: "list-member -w -n worker1"
got=1 expected=1
wait process dm-worker.test exit...
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 17:21:57 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/validator_basic/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
wait process dm-worker.test exit...
check diff successfully
check log contain failed 1-th time, retry later
wait process dm-master.test exit...
process dm-master.test already exit
wait process dm-worker3 exit...
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 17:21:58 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/new_relay/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
check diff failed at last
dmctl test cmd: "stop-task test"
wait process dm-worker.test exit...
[Sat May 4 17:21:59 CST 2024] <<<<<< finish DM-119 pessimistic >>>>>>
[Sat May 4 17:21:59 CST 2024] <<<<<< start DM-119 optimistic >>>>>>
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 17:21:59 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/validator_basic/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl4/conf/double-source-optimistic.yaml --remove-meta"
wait process dm-worker.test exit...
check diff successfully
dmctl test cmd: "stop-task test"
[Sat May 4 17:21:59 CST 2024] <<<<<< finish DM-010 >>>>>>
wait process dm-worker3 exit...
process dm-worker3 already exit
dmctl test cmd: "list-member -w -n worker1"
got=1 expected=1
got=1 expected=1
dmctl test cmd: "query-status load_task1"
got=1 expected=1
dmctl test cmd: "query-status load_task2"
wait for rpc addr 127.0.0.1:8261 alive the 2-th time
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 17:22:00 CST 2024] <<<<<< test case only_dml success! >>>>>>
start running case: [openapi] script: [/home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/openapi/run.sh]
Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/openapi/run.sh...
Verbose mode = false
got=1 expected=1
dmctl test cmd: "list-member -w -n worker1"
got=1 expected=1
got=0 expected=1
command: list-member -w -n worker1 "source": "mysql-replica-01" count: 0 != expected: 1, failed the 0-th time, will retry again
0 dm-master alive
0 dm-worker alive
0 dm-syncer alive
process dm-master.test already exit
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 17:22:00 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/openapi/conf/dm-master1.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
rpc addr 127.0.0.1:8262 is alive
dmctl test cmd: "operate-source create /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/validator_basic/conf/source1.yaml"
[Sat May 4 17:22:00 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/validator_basic/conf/dm-worker2.toml >>>>>>
kill dm-master1
check diff successfully
[Sat May 4 17:22:00 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/duplicate_event/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
wait process dm-worker.test exit...
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
[Sat May 4 17:22:00 CST 2024] <<<<<< start DM-011 >>>>>>
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 17:22:00 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/new_relay/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
dmctl test cmd: "query-status test"
got=2 expected=2
check diff failed 1-th time, retry later
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl1_1/conf/double-source-pessimistic.yaml --remove-meta"
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/duplicate_event/source2.yaml"
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/validator_basic/conf/dm-task-standalone-no-validator.yaml --remove-meta"
wait for rpc addr 127.0.0.1:8261 alive the 2-th time
wait process dm-master.test exit...
[Sat May 4 17:22:01 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/full_mode/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
rpc addr 127.0.0.1:8262 is alive
dmctl test cmd: "operate-source create /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/new_relay/conf/source1.yaml"
dmctl test cmd: "start-relay -s mysql-replica-01 worker1"
use sync_diff_inspector to check increment data
check diff successfully
check diff successfully
[Sat May 4 17:22:01 CST 2024] <<<<<< finish test_pause_task >>>>>>
[Sat May 4 17:22:01 CST 2024] <<<<<< start test_multi_task_reduce_and_restart_worker >>>>>>
[Sat May 4 17:22:01 CST 2024] <<<<<< start test_multi_task_running >>>>>>
3 dm-master alive
5 dm-worker alive
0 dm-syncer alive
wait process dm-master.test exit...
dmctl test cmd: "query-status test"
got=2 expected=2
dmctl test cmd: "stop-task test"
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 17:22:02 CST 2024] <<<<<< START DM-MASTER on port 8361, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/openapi/conf/dm-master2.toml >>>>>>
wait for rpc addr 127.0.0.1:8361 alive the 1-th time
[Sat May 4 17:22:02 CST 2024] <<<<<< finish DM-011 >>>>>>
wait process dm-master.test exit...
process dm-master.test already exit
got=1 expected=1
got=0 expected=1
command: list-member -w -n worker1 "source": "mysql-replica-01" count: 0 != expected: 1, failed the 1-th time, will retry again
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/duplicate_event/conf/dm-task-relay.yaml --remove-meta"
dmctl test cmd: "config source mysql-replica-01"
got=1 expected=1
got=1 expected=1
wait for rpc addr 127.0.0.1:8261 alive the 2-th time
wait process dm-master.test exit...
check diff failed 2-th time, retry later
got=1 expected=1
dmctl test cmd: "query-status -s mysql-replica-01"
got=2 expected=2
got=1 expected=1
[Sat May 4 17:22:03 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/new_relay/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
[Sat May 4 17:22:03 CST 2024] <<<<<< start DM-012 >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl1_1/conf/double-source-pessimistic.yaml --remove-meta"
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 17:22:03 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/full_mode/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
check diff successfully
dmctl test cmd: "start-relay -s mysql-replica-02 worker2"
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "operate-source create /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/new_relay/conf/source2.yaml"
wait process dm-master.test exit...
start check_worker_ungraceful_stop_with_retry times: 0
got=1 expected=1
got=0 expected=1
command: list-member -w -n worker1 "source": "mysql-replica-01" count: 0 != expected: 1, failed the 2-th time, will retry again
dmctl test cmd: "query-status test"
got=1 expected=1
check diff successfully
dmctl test cmd: "validation status test"
got=1 expected=1
dmctl test cmd: "validation start test"
dmctl test cmd: "query-status test"
dmctl test cmd: "validation status test"
got=1 expected=1
got=2 expected=2
dmctl test cmd: "query-status -s mysql-replica-02"
got=1 expected=1
got=2 expected=2
dmctl test cmd: "stop-task test"
check diff failed 3-th time, retry later
check diff successfully
binlog_pos: 2356 relay_log_size: 2356
============== run_with_prepared_source_config success ===================
1 dm-master alive
2 dm-worker alive
0 dm-syncer alive
rpc addr 127.0.0.1:8262 is alive
[Sat May 4 17:22:05 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/full_mode/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
wait process dm-master.test exit...
start check_worker_ungraceful_stop_with_retry times: 1
[Sat May 4 17:22:05 CST 2024] <<<<<< finish DM-012 >>>>>>
dmctl test cmd: "start-relay -s mysql-replica-02 worker2"
wait process dm-master.test exit...
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/full_mode/source1.yaml"
wait process dm-master.test exit...
[Sat May 4 17:22:06 CST 2024] <<<<<< start DM-013 >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl1_1/conf/single-source-optimistic.yaml --remove-meta"
dmctl test cmd: "query-status -s mysql-replica-02"
got=2 expected=2
got=1 expected=1
[Sat May 4 17:22:06 CST 2024] <<<<<< START DM-WORKER on port 8264, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/new_relay/conf/dm-worker3.toml >>>>>>
wait for rpc addr 127.0.0.1:8264 alive the 1-th time
start check_worker_ungraceful_stop_with_retry times: 2
check_worker_ungraceful_stop_with_retry success after retry: 2
after ungraceful exit data in source count: 31 data in tidb count: 17
ungraceful stop test success
[Sat May 4 17:22:06 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/checkpoint_transaction/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
dmctl test cmd: "validation status test"
got=0 expected=1
command: validation status test "processedRowsStatus": "insert\/update\/delete: 2\/1\/1" count: 0 != expected: 1, failed the 0-th time, will retry again
got=1 expected=1
got=0 expected=1
command: list-member -w -n worker1 "source": "mysql-replica-01" count: 0 != expected: 1, failed the 3-th time, will retry again
check diff failed 4-th time, retry later
dmctl test cmd: "operate-source create /tmp/dm_test/full_mode/source2.yaml"
wait process dm-master.test exit...
wait process dm-master.test exit...
process dm-master.test already exit
dmctl test cmd: "query-status test"
got=1 expected=1
rpc addr 127.0.0.1:8264 is alive
dmctl test cmd: "start-relay -s mysql-replica-02 worker3"
wait for rpc addr 127.0.0.1:8261 alive the 2-th time
wait process dm-master.test exit...
check diff successfully
dmctl test cmd: "stop-task test"
[Sat May 4 17:22:08 CST 2024] <<<<<< finish DM-013 >>>>>>
wait process dm-worker.test exit...
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/full_mode/conf/dm-task.yaml --remove-meta"
got=1 expected=1
got=0 expected=1
command: validation status test pendingRowsStatus": "insert\/update\/delete: 0\/0\/0 count: 0 != expected: 1, failed the 1-th time, will retry again
got=1 expected=1
got=0 expected=1
command: list-member -w -n worker1 "source": "mysql-replica-01" count: 0 != expected: 1, failed the 4-th time, will retry again
dmctl test cmd: "query-status -s mysql-replica-02"
got=3 expected=3
got=1 expected=1
got=1 expected=1
dmctl test cmd: "list-member -n worker3"
rpc addr 127.0.0.1:8261 is alive
wait process dm-master.test exit...
[Sat May 4 17:22:09 CST 2024] <<<<<< start DM-014 >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl1_1/conf/single-source-optimistic.yaml --remove-meta"
wait process dm-worker.test exit...
check diff failed 5-th time, retry later
dmctl test cmd: "query-status test"
got=2 expected=2
check diff successfully
check dump files have been cleaned
ls: cannot access /tmp/dm_test/full_mode/worker1/dumped_data.test: No such file or directory
worker1 auto removed dump files
ls: cannot access /tmp/dm_test/full_mode/worker2/dumped_data.test: No such file or directory
worker2 auto removed dump files
1 dm-master alive
2 dm-worker alive
0 dm-syncer alive
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 17:22:10 CST 2024] <<<<<< test case duplicate_event success! >>>>>>
start running case: [expression_filter] script: [/home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/expression_filter/run.sh]
Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/expression_filter/run.sh...
Verbose mode = false
0 dm-master alive
0 dm-worker alive
0 dm-syncer alive
process dm-master.test already exit
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 17:22:10 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/expression_filter/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
wait process dm-worker.test exit...
wait process dm-master.test exit...
process dm-master.test already exit
process dm-worker.test already exit
process dm-syncer.test already exit
clean source table
dmctl test cmd: "query-status test"
got=1 expected=1
got=1 expected=1
got=1 expected=1
dmctl test cmd: "list-member -w -n worker2"
run tidb sql failed 1-th time, retry later
got=1 expected=1
got=1 expected=1
dmctl test cmd: "query-status"
got=3 expected=3
got=1 expected=1
[Sat May 4 17:22:11 CST 2024] <<<<<< START DM-WORKER on port 8264, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/lightning_load_task/conf/dm-worker3.toml >>>>>>
wait for rpc addr 127.0.0.1:8264 alive the 1-th time
got=1 expected=1
got=1 expected=1
got=1 expected=1
check diff successfully
--> start validator on the fly, validate from current syncer progress(gtid=true, relay=false)
wait process dm-master.test exit...
wait process dm-worker.test exit...
process dm-worker.test already exit
check diff failed 6-th time, retry later
wait for rpc addr 127.0.0.1:8261 alive the 2-th time
1 dm-master alive
2 dm-worker alive
0 dm-syncer alive
wait process dm-master.test exit...
process dm-master.test already exit
wait process dm-master.test exit...
check diff failed 1-th time, retry later
import prepare data
start DM worker and master
[Sat May 4 17:22:12 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/ha_cases2/conf/dm-master1.toml >>>>>>
[Sat May 4 17:22:12 CST 2024] <<<<<< START DM-MASTER on port 8361, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/ha_cases2/conf/dm-master2.toml >>>>>>
[Sat May 4 17:22:12 CST 2024] <<<<<< START DM-MASTER on port 8461, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/ha_cases2/conf/dm-master3.toml >>>>>>
rpc addr 127.0.0.1:8264 is alive
dmctl test cmd: "list-member -w -n worker2"
got=1 expected=1
got=1 expected=1
dmctl test cmd: "query-status"
got=3 expected=4
command: query-status "taskStatus": "Running" count: 3 != expected: 4, failed the 0-th time, will retry again
wait for rpc addr 127.0.0.1:8361 alive the 2-th time
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 17:22:12 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/expression_filter/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
rpc addr 127.0.0.1:8361 is alive
[Sat May 4 17:22:12 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/openapi/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
dmctl test cmd: "stop-task test"
wait process dm-master.test exit...
wait process dm-worker.test exit...
wait process dm-master.test exit...
check diff failed 7-th time, retry later
[Sat May 4 17:22:13 CST 2024] <<<<<< finish DM-014 >>>>>>
process dm-master.test already exit
[Sat May 4 17:22:13 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/new_relay/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
rpc addr 127.0.0.1:8261 is alive
rpc addr 127.0.0.1:8361 is alive
rpc addr 127.0.0.1:8262 is alive
dmctl test cmd: "operate-source create /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/expression_filter/conf/source1.yaml"
rpc addr 127.0.0.1:8461 is alive
operate mysql config to worker
[Sat May 4 17:22:13 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/ha_cases2/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/expression_filter/conf/dm-task2.yaml"
wait process dm-master.test exit...
process dm-master.test already exit
rpc addr 127.0.0.1:8262 is alive
[Sat May 4 17:22:13 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/openapi/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
check diff failed 2-th time, retry later
[Sat May 4 17:22:14 CST 2024] <<<<<< start DM-015 >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl1_1/conf/single-source-pessimistic.yaml --remove-meta"
got=3 expected=4
command: query-status "taskStatus": "Running" count: 3 != expected: 4, failed the 1-th time, will retry again
rpc addr 127.0.0.1:8263 is alive
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>START TEST OPENAPI: RELAY
('create_source1_success resp=', {u'enable_gtid': False, u'enable': True, u'source_name': u'mysql-01', u'host': u'127.0.0.1', u'user': u'root', u'security': None, u'password': u'123456', u'port': 3306})
rpc addr 127.0.0.1:8262 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/ha_cases2/source1.yaml"
dmctl test cmd: "query-status test"
got=1 expected=1
got=1 expected=1
dmctl test cmd: "stop-task test"
dmctl test cmd: "query-status -s mysql-01"
got=2 expected=2
got=1 expected=1
wait for rpc addr 127.0.0.1:8261 alive the 2-th time
wait process dm-worker.test exit...
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/expression_filter/conf/dm-task2.yaml "
('enable_relay_failed resp=', {u'error_code': 46004, u'error_msg': u'[code=46004:class=scheduler:scope=internal:level=medium], Message: dm-worker with name [no-worker] not exists'})
dmctl test cmd: "query-status -s mysql-01"
dmctl test cmd: "query-status test"
got=1 expected=1
check log contain failed 1-th time, retry later
got=1 expected=1
got=1 expected=1
('get_source_status_failed resp=', {u'error_code': 46008, u'error_msg': u'[code=46008:class=scheduler:scope=internal:level=medium], Message: source config with ID no-mysql not exists'})
('get_source_status_success resp=', {u'total': 1, u'data': [{u'relay_status': {u'relay_dir': u'a81cef20-09f6-11ef-a962-9e663be6c40c.000001', u'relay_catch_up_master': True, u'master_binlog_gtid': u'a81cef20-09f6-11ef-a962-9e663be6c40c:1-2', u'master_binlog': u'(dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001, 488)', u'relay_binlog_gtid': u'', u'stage': u'Running'}, u'worker_name': u'worker1', u'source_name': u'mysql-01'}]})
check diff failed 8-th time, retry later
('get_source_status_success_with_relay resp=', {u'total': 1, u'data': [{u'relay_status': {u'relay_dir': u'a81cef20-09f6-11ef-a962-9e663be6c40c.000001', u'relay_catch_up_master': True, u'master_binlog_gtid': u'a81cef20-09f6-11ef-a962-9e663be6c40c:1-2', u'master_binlog': u'(dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001, 488)', u'relay_binlog_gtid': u'', u'stage': u'Running'}, u'worker_name': u'worker1', u'source_name': u'mysql-01'}]})
('disable_relay_failed resp=', {u'error_code': 46004, u'error_msg': u'[code=46004:class=scheduler:scope=internal:level=medium], Message: dm-worker with name [no-worker] not exists'})
[Sat May 4 17:22:16 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/ha_cases2/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 17:22:16 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/new_relay/conf/dm-worker1.toml >>>>>>
wait process dm-worker.test exit...
dmctl test cmd: "query-status -s mysql-01"
wait process dm-worker.test exit...
[Sat May 4 17:22:16 CST 2024] <<<<<< START DM-WORKER on port 8264, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/new_relay/conf/dm-worker3.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
got=1 expected=1
got=1 expected=1
('get_source_status_success_no_relay resp=', {u'total': 1, u'data': [{u'worker_name': u'worker1', u'source_name': u'mysql-01'}]})
check diff failed 3-th time, retry later
('list_source_with_status_success resp=', {u'total': 1, u'data': [{u'enable_gtid': False, u'enable': True, u'source_name': u'mysql-01', u'security': None, u'relay_config': {u'relay_binlog_name': u'', u'relay_dir': u'relay-dir', u'relay_binlog_gtid': u'', u'enable_relay': False}, u'purge': {u'remain_space': 15, u'expires': 0, u'interval': 3600}, u'host': u'127.0.0.1', u'user': u'root', u'task_name_list': [], u'flavor': u'mysql', u'password': u'******', u'port': 3306, u'status_list': [{u'relay_status': {u'relay_dir': u'a81cef20-09f6-11ef-a962-9e663be6c40c.000001', u'relay_catch_up_master': True, u'master_binlog_gtid': u'a81cef20-09f6-11ef-a962-9e663be6c40c:1-2', u'master_binlog': u'(dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001, 488)', u'relay_binlog_gtid': u'a81cef20-09f6-11ef-a962-9e663be6c40c:1-2', u'stage': u'Running'}, u'worker_name': u'worker1', u'source_name': u'mysql-01'}, {u'relay_status': {u'relay_dir': u'', u'relay_catch_up_master': False, u'master_binlog_gtid': u'a81cef20-09f6-11ef-a962-9e663be6c40c:1-2', u'master_binlog': u'(dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001, 488)', u'relay_binlog_gtid': u'', u'stage': u'Running'}, u'worker_name': u'worker2', u'source_name': u'mysql-01'}]}]})
('get_source_status_success resp=', {u'total': 2, u'data': [{u'relay_status': {u'relay_dir': u'a81cef20-09f6-11ef-a962-9e663be6c40c.000001', u'relay_catch_up_master': True, u'master_binlog_gtid': u'a81cef20-09f6-11ef-a962-9e663be6c40c:1-2', u'master_binlog': u'(dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001, 488)', u'relay_binlog_gtid': u'', u'stage': u'Running'}, u'worker_name': u'worker2', u'source_name': u'mysql-01'}, {u'relay_status': {u'relay_dir': u'a81cef20-09f6-11ef-a962-9e663be6c40c.000001', u'relay_catch_up_master': True, u'master_binlog_gtid': u'a81cef20-09f6-11ef-a962-9e663be6c40c:1-2', u'master_binlog': u'(dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001, 488)', u'relay_binlog_gtid': u'a81cef20-09f6-11ef-a962-9e663be6c40c:1-2', u'stage': u'Running'}, u'worker_name': u'worker1', u'source_name': u'mysql-01'}]})
got=3 expected=4
command: query-status "taskStatus": "Running" count: 3 != expected: 4, failed the 2-th time, will retry again
dmctl test cmd: "query-status test"
got=2 expected=2
got=0 expected=1
command: query-status test "synced": true count: 0 != expected: 1, failed the 0-th time, will retry again
('get_source_status_success_with_relay resp=', {u'total': 2, u'data': [{u'relay_status': {u'relay_dir': u'a81cef20-09f6-11ef-a962-9e663be6c40c.000001', u'relay_catch_up_master': True, u'master_binlog_gtid': u'a81cef20-09f6-11ef-a962-9e663be6c40c:1-2', u'master_binlog': u'(dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001, 488)', u'relay_binlog_gtid': u'', u'stage': u'Running'}, u'worker_name': u'worker2', u'source_name': u'mysql-01'}, {u'relay_status': {u'relay_dir': u'a81cef20-09f6-11ef-a962-9e663be6c40c.000001', u'relay_catch_up_master': True, u'master_binlog_gtid': u'a81cef20-09f6-11ef-a962-9e663be6c40c:1-2', u'master_binlog': u'(dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001, 488)', u'relay_binlog_gtid': u'a81cef20-09f6-11ef-a962-9e663be6c40c:1-2', u'stage': u'Running'}, u'worker_name': u'worker1', u'source_name': u'mysql-01'}]})
('get_source_status_success_with_relay resp=', {u'total': 2, u'data': [{u'relay_status': {u'relay_dir': u'a81cef20-09f6-11ef-a962-9e663be6c40c.000001', u'relay_catch_up_master': True, u'master_binlog_gtid': u'a81cef20-09f6-11ef-a962-9e663be6c40c:1-2', u'master_binlog': u'(dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001, 488)', u'relay_binlog_gtid': u'', u'stage': u'Running'}, u'worker_name': u'worker2', u'source_name': u'mysql-01'}, {u'relay_status': {u'relay_dir': u'a81cef20-09f6-11ef-a962-9e663be6c40c.000001', u'relay_catch_up_master': True, u'master_binlog_gtid': u'a81cef20-09f6-11ef-a962-9e663be6c40c:1-2', u'master_binlog': u'(dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001, 488)', u'relay_binlog_gtid': u'a81cef20-09f6-11ef-a962-9e663be6c40c:1-2', u'stage': u'Running'}, u'worker_name': u'worker1', u'source_name': u'mysql-01'}]})
dmctl test cmd: "query-status -s mysql-01"
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 17:22:17 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/validator_basic/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 17:22:17 CST 2024] <<<<<< test case full_mode success! >>>>>>
start running case: [gbk] script: [/home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/gbk/run.sh]
Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/gbk/run.sh...
Verbose mode = false
0 dm-master alive
0 dm-worker alive
0 dm-syncer alive
process dm-master.test already exit
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 17:22:17 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/gbk/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
got=1 expected=1
got=1 expected=1
rpc addr 127.0.0.1:8262 is alive
rpc addr 127.0.0.1:8264 is alive
[Sat May 4 17:22:17 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/new_relay/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/ha_cases2/source2.yaml"
delete_source_success
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>TEST OPENAPI: RELAY SUCCESS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>START TEST OPENAPI: SOURCE
('create_source1_success resp=', {u'enable_gtid': False, u'enable': True, u'source_name': u'mysql-01', u'host': u'127.0.0.1', u'user': u'root', u'security': None, u'password': u'123456', u'port': 3306})
check diff failed 9-th time, retry later
('create_source_failed resp=', {u'error_code': 10001, u'error_msg': u'[code=10001:class=database:scope=upstream:level=high], Message: database driver error, RawCause: dial tcp :0: connect: connection refused, Workaround: Please check the database connection and the database config in configuration file.'})
('update_source1_without_password_success resp=', {u'enable_gtid': False, u'enable': True, u'source_name': u'mysql-01', u'host': u'127.0.0.1', u'user': u'root', u'security': None, u'password': None, u'port': 3306})
('list_source_by_openapi_success resp=', {u'total': 1, u'data': [{u'enable_gtid': False, u'enable': True, u'source_name': u'mysql-01', u'security': None, u'relay_config': {u'relay_binlog_name': u'', u'relay_dir': u'relay-dir', u'relay_binlog_gtid': u'', u'enable_relay': False}, u'purge': {u'remain_space': 15, u'expires': 0, u'interval': 3600}, u'host': u'127.0.0.1', u'user': u'root', u'flavor': u'mysql', u'password': u'******', u'port': 3306}]})
('list_source_with_status_success resp=', {u'total': 1, u'data': [{u'enable_gtid': False, u'enable': True, u'source_name': u'mysql-01', u'security': None, u'relay_config': {u'relay_binlog_name': u'', u'relay_dir': u'relay-dir', u'relay_binlog_gtid': u'', u'enable_relay': False}, u'purge': {u'remain_space': 15, u'expires': 0, u'interval': 3600}, u'host': u'127.0.0.1', u'user': u'root', u'task_name_list': [], u'flavor': u'mysql', u'password': u'******', u'port': 3306, u'status_list': [{u'worker_name': u'worker1', u'source_name': u'mysql-01'}]}]})
dmctl test cmd: "query-status -s mysql-01"
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 17:22:18 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/gbk/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
dmctl test cmd: "stop-task test"
[Sat May 4 17:22:18 CST 2024] <<<<<< finish DM-015 >>>>>>
got=1 expected=1
('get_source_schemas_and_tables_success schema_resp=', [u'information_schema', u'ha_test', u'import_v10x', u'mysql', u'only_dml', u'openapi', u'performance_schema', u'sharding22', u'sys'])
('get_source_schemas_and_tables_success table_resp=', [u't1'])
[Sat May 4 17:22:18 CST 2024] <<<<<< START DM-WORKER on port 8264, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/ha_cases2/conf/dm-worker3.toml >>>>>>
wait for rpc addr 127.0.0.1:8264 alive the 1-th time
check diff failed 4-th time, retry later
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 17:22:18 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/validator_basic/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
('get_source_schemas_and_tables_success schema_resp=', [u'information_schema', u'db-name', u'ha_test', u'import_v10x', u'mysql', u'only_dml', u'openapi', u'performance_schema', u'sharding22', u'sys'])
('get_source_schemas_and_tables_success table_resp=', [u't1'])
delete_source_success
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "query-status -s mysql-replica-01"
got=2 expected=2
got=1 expected=1
dmctl test cmd: "query-status -s mysql-replica-02"
got=3 expected=3
got=1 expected=1
got=1 expected=1
dmctl test cmd: "list-member --worker"
>>>>>>>>>>>>>>>>>>>>>>>>>>test test_restart_relay_status passed
1 dm-master alive
3 dm-worker alive
0 dm-syncer alive
('list_source_by_openapi_success resp=', {u'total': 0, u'data': []})
('delete_source_failed msg=', {u'error_code': 46008, u'error_msg': u'[code=46008:class=scheduler:scope=internal:level=medium], Message: source config with ID mysql-01 not exists'})
('list_source_with_reverse resp=', {u'total': 0, u'data': []})
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>TEST OPENAPI: SOURCE SUCCESS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>START TEST OPENAPI: SHARD TASK
('create_source1_success resp=', {u'enable_gtid': False, u'enable': True, u'source_name': u'mysql-01', u'host': u'127.0.0.1', u'user': u'root', u'security': None, u'password': u'123456', u'port': 3306})
('list_source_by_openapi_success resp=', {u'total': 1, u'data': [{u'enable_gtid': False, u'enable': True, u'source_name': u'mysql-01', u'security': None, u'relay_config': {u'relay_binlog_name': u'', u'relay_dir': u'relay-dir', u'relay_binlog_gtid': u'', u'enable_relay': False}, u'purge': {u'remain_space': 15, u'expires': 0, u'interval': 3600}, u'host': u'127.0.0.1', u'user': u'root', u'flavor': u'mysql', u'password': u'******', u'port': 3306}]})
[Sat May 4 17:22:19 CST 2024] <<<<<< start DM-016 >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl1_1/conf/single-source-pessimistic.yaml --remove-meta"
('get_source_status_success resp=', {u'total': 1, u'data': [{u'worker_name': u'worker1', u'source_name': u'mysql-01'}]})
('create_source1_success resp=', {u'enable_gtid': False, u'enable': True, u'source_name': u'mysql-02', u'host': u'127.0.0.1', u'user': u'root', u'security': None, u'password': u'123456', u'port': 3307})
got=3 expected=4
command: query-status "taskStatus": "Running" count: 3 != expected: 4, failed the 3-th time, will retry again
got=2 expected=2
got=1 expected=1
dmctl test cmd: "stop-task test"
ls: cannot access /tmp/dm_test/expression_filter/worker1/schema-tracker*: No such file or directory
schema tracker path has been cleaned
1 dm-master alive
1 dm-worker alive
0 dm-syncer alive
rpc addr 127.0.0.1:8262 is alive
dmctl test cmd: "operate-source create /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/validator_basic/conf/source1.yaml"
rpc addr 127.0.0.1:8262 is alive
[Sat May 4 17:22:19 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/gbk/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
('list_source_by_openapi_success resp=', {u'total': 2, u'data': [{u'enable_gtid': False, u'enable': True, u'source_name': u'mysql-01', u'security': None, u'relay_config': {u'relay_binlog_name': u'', u'relay_dir': u'relay-dir', u'relay_binlog_gtid': u'', u'enable_relay': False}, u'purge': {u'remain_space': 15, u'expires': 0, u'interval': 3600}, u'host': u'127.0.0.1', u'user': u'root', u'flavor': u'mysql', u'password': u'******', u'port': 3306}, {u'enable_gtid': False, u'enable': True, u'source_name': u'mysql-02', u'security': None, u'relay_config': {u'relay_binlog_name': u'', u'relay_dir': u'relay-dir', u'relay_binlog_gtid': u'', u'enable_relay': False}, u'purge': {u'remain_space': 15, u'expires': 0, u'interval': 3600}, u'host': u'127.0.0.1', u'user': u'root', u'flavor': u'mysql', u'password': u'******', u'port': 3307}]})
('get_source_status_success resp=', {u'total': 1, u'data': [{u'worker_name': u'worker2', u'source_name': u'mysql-02'}]})
('create_task_failed resp=', {u'error': u'error in openapi3filter.RequestError: request body has an error: doesn\'t match the schema: Error at "/task/shard_mode": value is not one of the allowed values'})
check diff failed 10-th time, retry later
rpc addr 127.0.0.1:8264 is alive
[Sat May 4 17:22:19 CST 2024] <<<<<< START DM-WORKER on port 18262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/ha_cases2/conf/dm-worker4.toml >>>>>>
wait for rpc addr 127.0.0.1:18262 alive the 1-th time
('create_shard_task_success resp=', {u'check_result': u'pre-check is passed. ', u'task': {u'binlog_filter_rule': {u'rule-2': {u'ignore_sql': [u'alter table openapi.t add column aaa int;']}, u'rule-1': {u'ignore_event': [u'delete']}}, u'name': u'test-shard', u'enhance_online_schema_change': True, u'on_duplicate': u'error', u'table_migrate_rule': [{u'source': {u'table': u'*', u'source_name': u'mysql-01', u'schema': u'openapi'}, u'binlog_filter_rule': [u'rule-1'], u'target': {u'table': u't', u'schema': u'openapi'}}, {u'source': {u'table': u'*', u'source_name': u'mysql-02', u'schema': u'openapi'}, u'binlog_filter_rule': [u'rule-2'], u'target': {u'table': u't', u'schema': u'openapi'}}], u'target_config': {u'user': u'root', u'host': u'127.0.0.1', u'password': u'', u'port': 4000, u'security': None}, u'meta_schema': u'dm-meta', u'shard_mode': u'pessimistic', u'task_mode': u'all', u'source_config': {u'incr_migrate_conf': {u'repl_batch': 100, u'repl_threads': 16}, u'full_migrate_conf': {u'import_threads': 16, u'data_dir': u'./exported_data', u'export_threads': 4, u'consistency': u'auto'}, u'source_conf': [{u'source_name': u'mysql-01'}, {u'source_name': u'mysql-02'}]}}})
dmctl test cmd: "query-status test-shard"
wait process dm-master.test exit...
wait process dm-master.test exit...
[Sat May 4 17:22:20 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/validator_basic/conf/dm-worker2.toml >>>>>>
check diff failed 5-th time, retry later
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/gbk/source1.yaml"
dmctl test cmd: "query-status test"
got=1 expected=1
dmctl test cmd: "stop-task test"
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
rpc addr 127.0.0.1:18262 is alive
[Sat May 4 17:22:20 CST 2024] <<<<<< START DM-WORKER on port 18263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/ha_cases2/conf/dm-worker5.toml >>>>>>
wait for rpc addr 127.0.0.1:18263 alive the 1-th time
dmctl test cmd: "operate-source create /tmp/dm_test/gbk/source2.yaml"
[Sat May 4 17:22:20 CST 2024] <<<<<< finish DM-016 >>>>>>
got=2 expected=2
dmctl test cmd: "query-status test-shard"
got=2 expected=2
check diff successfully
('get_task_status_failed resp=', {u'error_code': 46018, u'error_msg': u'[code=46018:class=scheduler:scope=internal:level=medium], Message: task with name not a task name not exist, Workaround: Please use `query-status` command to see tasks.'})
wait process dm-master.test exit...
process dm-master.test already exit
got=3 expected=4
command: query-status "taskStatus": "Running" count: 3 != expected: 4, failed the 4-th time, will retry again
wait process dm-master.test exit...
process dm-master.test already exit
('get_illegal_char_task_status_failed resp=', {u'error_code': 46018, u'error_msg': u'[code=46018:class=scheduler:scope=internal:level=medium], Message: task with name t-\xcb!s`t not exist, Workaround: Please use `query-status` command to see tasks.'})
('get_task_status_success resp=', {u'total': 2, u'data': [{u'source_name': u'mysql-01', u'name': u'test-shard', u'unresolved_ddl_lock_id': u'', u'sync_status': {u'binlog_type': u'remote', u'blocking_ddls': None, u'recent_tps': 0, u'total_events': 6, u'master_binlog_gtid': u'a81cef20-09f6-11ef-a962-9e663be6c40c:1-15', u'seconds_behind_master': 0, u'master_binlog': u'(dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001, 3103)', u'syncer_binlog_gtid': u'a81cef20-09f6-11ef-a962-9e663be6c40c:1-12', u'synced': False, u'total_tps': 0, u'unresolved_groups': [{u'synced': None, u'first_location': u'', u'unsynced': [u"this DM-worker doesn't receive any shard DDL of this group"], u'target': u'`openapi`.`t`', u'ddl_list': None}], u'syncer_binlog': u'(dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001, 2344)'}, u'worker_name': u'worker1', u'unit': u'Sync', u'stage': u'Running'}, {u'source_name': u'mysql-02', u'name': u'test-shard', u'unresolved_ddl_lock_id': u'', u'sync_status': {u'binlog_type': u'remote', u'blocking_ddls': [u'ALTER TABLE `openapi`.`t` ADD COLUMN `aaa` INT'], u'recent_tps': 0, u'total_events': 6, u'master_binlog_gtid': u'a872c033-09f6-11ef-951c-9e663be6c40c:1-11', u'seconds_behind_master': 0, u'master_binlog': u'(dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001, 2511)', u'syncer_binlog_gtid': u'a872c033-09f6-11ef-951c-9e663be6c40c:1-10', u'synced': False, u'total_tps': 0, u'unresolved_groups': [{u'synced': [u'`openapi`.`t`'], u'first_location': u'position: (dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001, 2384), gtid-set: a872c033-09f6-11ef-951c-9e663be6c40c:1-10', u'unsynced': None, u'target': u'`openapi`.`t`', u'ddl_list': [u'ALTER TABLE `openapi`.`t` ADD COLUMN `aaa` INT']}], u'syncer_binlog': u'(dm-it-10a4c9e9-7daf-4005-9282-b97746844222-6pfr6-h0vm0-bin.000001, 2307)'}, u'worker_name': u'worker2', u'unit': u'Sync', u'stage': u'Running'}]})
check diff failed at last
dmctl test cmd: "stop-task test"
('get_task_list resp=', {u'total': 1, u'data': [{u'strict_optimistic_shard_mode': False, u'binlog_filter_rule': {u'mysql-01-filter-rule-0': {u'ignore_event': [u'delete']}, u'mysql-02-filter-rule-0': {u'ignore_sql': [u'alter table openapi.t add column aaa int;']}}, u'name': u'test-shard', u'enhance_online_schema_change': True, u'on_duplicate': u'replace', u'table_migrate_rule': [{u'source': {u'table': u'*', u'source_name': u'mysql-02', u'schema': u'openapi'}, u'binlog_filter_rule': [u'mysql-02-filter-rule-0'], u'target': {u'table': u't', u'schema': u'openapi'}}, {u'source': {u'table': u'*', u'source_name': u'mysql-01', u'schema': u'openapi'}, u'binlog_filter_rule': [u'mysql-01-filter-rule-0'], u'target': {u'table': u't', u'schema': u'openapi'}}], u'target_config': {u'user': u'root', u'host': u'127.0.0.1', u'password': u'', u'port': 4000, u'security': None}, u'meta_schema': u'dm-meta', u'shard_mode': u'pessimistic', u'task_mode': u'all', u'source_config': {u'incr_migrate_conf': {u'repl_batch': 100, u'repl_threads': 16}, u'full_migrate_conf': {u'import_threads': 16, u'data_dir': u'./exported_data', u'export_threads': 4, u'consistency': u'auto'}, u'source_conf': [{u'source_name': u'mysql-01'}, {u'source_name': u'mysql-02'}]}}]})
('get_task_list resp=', {u'total': 1, u'data': [{u'strict_optimistic_shard_mode': False, u'binlog_filter_rule': {u'mysql-01-filter-rule-0': {u'ignore_event': [u'delete']}, u'mysql-02-filter-rule-0': {u'ignore_sql': [u'alter table openapi.t add column aaa int;']}}, u'name': u'test-shard', u'enhance_online_schema_change': True, u'on_duplicate': u'replace', u'table_migrate_rule': [{u'source': {u'table': u'*', u'source_name': u'mysql-01', u'schema': u'openapi'}, u'binlog_filter_rule': [u'mysql-01-filter-rule-0'], u'target': {u'table': u't', u'schema': u'openapi'}}, {u'source': {u'table': u'*', u'source_name': u'mysql-02', u'schema': u'openapi'}, u'binlog_filter_rule': [u'mysql-02-filter-rule-0'], u'target': {u'table': u't', u'schema': u'openapi'}}], u'target_config': {u'user': u'root', u'host': u'127.0.0.1', u'password': u'', u'port': 4000, u'security': None}, u'meta_schema': u'dm-meta', u'shard_mode': u'pessimistic', u'task_mode': u'all', u'source_config': {u'incr_migrate_conf': {u'repl_batch': 100, u'repl_threads': 16}, u'full_migrate_conf': {u'import_threads': 16, u'data_dir': u'./exported_data', u'export_threads': 4, u'consistency': u'auto'}, u'source_conf': [{u'source_name': u'mysql-01'}, {u'source_name': u'mysql-02'}]}}]})
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/validator_basic/conf/dm-task-standalone-no-validator.yaml --remove-meta"
rpc addr 127.0.0.1:18263 is alive
start DM task
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/ha_cases2/conf/dm-task2.yaml "
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/ha_cases2/conf/dm-task.yaml "
[Sat May 4 17:22:21 CST 2024] <<<<<< start DM-017 >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl1_1/conf/single-source-pessimistic.yaml --remove-meta"
prepare data
[Sat May 4 17:22:21 CST 2024] <<<<<< finish DM-119 optimistic >>>>>>
delete_task_success
start task
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/gbk/conf/dm-task.yaml --remove-meta"
('get_task_list resp=', {u'total': 0, u'data': []})
delete_source_with_force_success
wait process dm-worker.test exit...
wait process dm-worker.test exit...
delete_source_with_force_success
('list_source_by_openapi_success resp=', {u'total': 0, u'data': []})
('get_task_list resp=', {u'total': 0, u'data': []})
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>TEST OPENAPI: SHARD TASK SUCCESS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>START TEST OPENAPI: MULTI TASK
Aborted by Jenkins Admin
Sending interrupt signal to process
Killing processes
('create_source1_success resp=', {u'enable_gtid': False, u'enable': True, u'source_name': u'mysql-01', u'host': u'127.0.0.1', u'user': u'root', u'security': None, u'password': u'123456', u'port': 3306})
('create_source1_success resp=', {u'enable_gtid': False, u'enable': True, u'source_name': u'mysql-02', u'host': u'127.0.0.1', u'user': u'root', u'security': None, u'password': u'123456', u'port': 3307})
kill finished with exit code 0
Sending interrupt signal to process
Killing processes
('list_source_by_openapi_success resp=', {u'total': 2, u'data': [{u'enable_gtid': False, u'enable': True, u'source_name': u'mysql-01', u'security': None, u'relay_config': {u'relay_binlog_name': u'', u'relay_dir': u'relay-dir', u'relay_binlog_gtid': u'', u'enable_relay': False}, u'purge': {u'remain_space': 15, u'expires': 0, u'interval': 3600}, u'host': u'127.0.0.1', u'user': u'root', u'flavor': u'mysql', u'password': u'******', u'port': 3306}, {u'enable_gtid': False, u'enable': True, u'source_name': u'mysql-02', u'security': None, u'relay_config': {u'relay_binlog_name': u'', u'relay_dir': u'relay-dir', u'relay_binlog_gtid': u'', u'enable_relay': False}, u'purge': {u'remain_space': 15, u'expires': 0, u'interval': 3600}, u'host': u'127.0.0.1', u'user': u'root', u'flavor': u'mysql', u'password': u'******', u'port': 3307}]})
kill finished with exit code 0
Sending interrupt signal to process
Killing processes
check diff failed 6-th time, retry later
restore config
make: *** [dm_integration_test_in_group] Terminated
/home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/validator_basic/run.sh: line 53: 8659 Terminated run_dm_ctl $WORK_DIR "127.0.0.1:$MASTER_PORT" "start-task $task_conf $remove_meta" "\"result\": true" 2 "\"source\": \"$SOURCE_ID1\"" 1
restore time_zone
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
script returned exit code 143
wait process dm-worker.test exit...
make: *** [dm_integration_test_in_group] Terminated
script returned exit code 143
dmctl test cmd: "query-status test"
('create_noshard_task_success resp=', {u'check_result': u'pre-check is passed. ', u'task': {u'source_config': {u'source_conf': [{u'source_name': u'mysql-01'}, {u'source_name': u'mysql-02'}]}, u'enhance_online_schema_change': True, u'on_duplicate': u'error', u'table_migrate_rule': [{u'source': {u'table': u'*', u'source_name': u'mysql-01', u'schema': u'openapi'}, u'target': {u'table': u'task1_target_table', u'schema': u'openapi'}}, {u'source': {u'table': u'*', u'source_name': u'mysql-02', u'schema': u'openapi'}, u'target': {u'table': u'task1_target_table', u'schema': u'openapi'}}], u'target_config': {u'user': u'root', u'host': u'127.0.0.1', u'password': u'', u'port': 4000, u'security': None}, u'meta_schema': u'dm-meta', u'task_mode': u'all', u'name': u'test-1'}})
dmctl test cmd: "query-status test-1"
got=2 expected=2
[Sat May 4 17:22:22 CST 2024] <<<<<< start DM-120 pessimistic >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl4/conf/double-source-pessimistic.yaml --remove-meta"
got=3 expected=4
command: query-status "taskStatus": "Running" count: 3 != expected: 4, failed the 5-th time, will retry again
kill finished with exit code 0
Sending interrupt signal to process
Killing processes
got=1 expected=1
check log contain failed 1-th time, retry later
check diff successfully
prepare incremental data
check incremental phase
check diff successfully
ERROR 1146 (42S02) at line 1: Table 'gbk.ddl1' doesn't exist
run tidb sql failed 1-th time, retry later
dmctl test cmd: "query-status test-1"
make: *** [dm_integration_test_in_group] Terminated
script returned exit code 143
kill finished with exit code 0
Sending interrupt signal to process
Killing processes
dmctl test cmd: "query-status test"
got=2 expected=2
got=2 expected=2
dmctl test cmd: "query-status test2"
got=2 expected=2
got=2 expected=2
use sync_diff_inspector to check full dump loader
wait process dm-worker.test exit...
make: *** [dm_integration_test_in_group] Terminated
script returned exit code 143
use sync_diff_inspector to check increment data
check diff successfully
[Sat May 4 17:22:19 CST 2024] <<<<<< finish test_isolate_master_and_worker >>>>>>
5 dm-master alive
5 dm-worker alive
0 dm-syncer alive
wait process dm-master.test exit...
wait process dm-master.test exit...
wait process dm-master.test exit...
wait process dm-master.test exit...
kill finished with exit code 0
Sending interrupt signal to process
Killing processes
check diff successfully
check diff successfully
flush logs to force rotate binlog file
apply increment data before restart dm-worker to ensure entering increment phase
make: *** [dm_integration_test_in_group] Terminated
script returned exit code 143
kill finished with exit code 0
Sending interrupt signal to process
Killing processes
make: *** [dm_integration_test_in_group] Terminated
script returned exit code 143
kill finished with exit code 0
Sending interrupt signal to process
Killing processes
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 17:22:24 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/expression_filter/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
kill finished with exit code 0
Sending interrupt signal to process
Killing processes
wait process dm-master.test exit...
make: *** [dm_integration_test_in_group] Terminated
script returned exit code 143
make: *** [dm_integration_test_in_group] Terminated
script returned exit code 143
kill finished with exit code 0
Sending interrupt signal to process
Killing processes
check diff failed 7-th time, retry later
make: *** [dm_integration_test_in_group] Terminated
script returned exit code 143
kill finished with exit code 0
Sending interrupt signal to process
Killing processes
wait process dm-master.test exit...
make: *** [dm_integration_test_in_group] Terminated
script returned exit code 143
kill finished with exit code 0
make: *** [dm_integration_test_in_group] Terminated
script returned exit code 143
[Pipeline] }
Cache not saved (inner-step execution failed)
[Pipeline] }
Cache not saved (inner-step execution failed)
[Pipeline] }
Cache not saved (inner-step execution failed)
[Pipeline] }
Cache not saved (inner-step execution failed)
[Pipeline] }
Cache not saved (inner-step execution failed)
[Pipeline] }
Cache not saved (inner-step execution failed)
[Pipeline] }
Cache not saved (inner-step execution failed)
[Pipeline] }
Cache not saved (inner-step execution failed)
[Pipeline] }
Cache not saved (inner-step execution failed)
[Pipeline] }
Cache not saved (inner-step execution failed)
[Pipeline] }
Cache not saved (inner-step execution failed)
[Pipeline] // cache
[Pipeline] // cache
[Pipeline] // cache
[Pipeline] // cache
[Pipeline] // cache
[Pipeline] // cache
[Pipeline] // cache
[Pipeline] // cache
[Pipeline] // cache
[Pipeline] // cache
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // cache
[Pipeline] }
[Pipeline] // dir
[Pipeline] // dir
[Pipeline] // dir
[Pipeline] // dir
[Pipeline] // dir
[Pipeline] // dir
[Pipeline] // dir
[Pipeline] // dir
[Pipeline] // dir
[Pipeline] // dir
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] // withCredentials
[Pipeline] // withCredentials
[Pipeline] // withCredentials
[Pipeline] // withCredentials
[Pipeline] // withCredentials
[Pipeline] // withCredentials
[Pipeline] // withCredentials
[Pipeline] // withCredentials
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // timeout
[Pipeline] // timeout
[Pipeline] // timeout
[Pipeline] // timeout
[Pipeline] // timeout
[Pipeline] // timeout
[Pipeline] // timeout
[Pipeline] // timeout
[Pipeline] // timeout
[Pipeline] // timeout
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] // container
[Pipeline] // container
[Pipeline] // container
[Pipeline] // container
[Pipeline] // container
[Pipeline] // container
[Pipeline] // container
[Pipeline] // container
[Pipeline] // container
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] // node
[Pipeline] // node
[Pipeline] // node
[Pipeline] // node
[Pipeline] // node
[Pipeline] // node
[Pipeline] // node
[Pipeline] // node
[Pipeline] // node
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] // podTemplate
[Pipeline] // podTemplate
[Pipeline] // podTemplate
[Pipeline] // podTemplate
[Pipeline] // podTemplate
[Pipeline] // podTemplate
[Pipeline] // podTemplate
[Pipeline] // podTemplate
[Pipeline] // podTemplate
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] }
Failed in branch Matrix - TEST_GROUP = 'G01'
[Pipeline] }
Failed in branch Matrix - TEST_GROUP = 'G02'
[Pipeline] }
Failed in branch Matrix - TEST_GROUP = 'G04'
[Pipeline] }
Failed in branch Matrix - TEST_GROUP = 'G05'
[Pipeline] }
Failed in branch Matrix - TEST_GROUP = 'G06'
[Pipeline] }
Failed in branch Matrix - TEST_GROUP = 'G07'
[Pipeline] }
Failed in branch Matrix - TEST_GROUP = 'G08'
[Pipeline] }
Failed in branch Matrix - TEST_GROUP = 'G09'
[Pipeline] }
Failed in branch Matrix - TEST_GROUP = 'G10'
[Pipeline] }
Failed in branch Matrix - TEST_GROUP = 'G11'
Click here to forcibly terminate running steps
[Pipeline] // stage
[Pipeline] }
Failed in branch Matrix - TEST_GROUP = 'G00'
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 17f40361-1fdf-471e-9086-c4c11b4f87df
Finished: ABORTED
![[Jenkins]](/jenkins/static/62b7ca6c/images/svgs/logo.svg){kind=logo}
