Skipping 847 KB..
Full LogTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTest
got=2 expected=2
dmctl test cmd: "binlog skip test -b dm-it-175e207b-2603-4dff-98a7-3fe52f932268-n5x6k-7hgzv-bin|000001.000001:29861"
dmctl test cmd: "query-status test"
wait process dm-master.test exit...
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 12:35:00 CST 2024] <<<<<< test case expression_filter success! >>>>>>
start running case: [extend_column] script: [/home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/extend_column/run.sh]
Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/extend_column/run.sh...
Verbose mode = false
0 dm-master alive
0 dm-worker alive
0 dm-syncer alive
process dm-master.test already exit
process dm-worker.test already exit
process dm-syncer.test already exit
running extend_column case with import_mode: sql
[Sat May 4 12:35:00 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/extend_column/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
start_task_with_condition success
dmctl test cmd: "query-status incremental_task_use_start_time"
got=2 expected=2
dmctl test cmd: "binlog skip test"
dmctl test cmd: "query-status test"
got=2 expected=2
got=3 expected=3
dmctl test cmd: "stop-task test"
delete_task_success
('get_task_list resp=', {u'total': 0, u'data': []})
rpc addr 127.0.0.1:8261 is alive
dmctl test cmd: "pause-task test"
got=3 expected=3
got=1 expected=1
got=1 expected=1
4 dm-master alive
2 dm-worker alive
0 dm-syncer alive
wait for rpc addr 127.0.0.1:8261 alive the 2-th time
[Sat May 4 12:35:01 CST 2024] <<<<<< start DM-106 optimistic >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl3_1/conf/double-source-optimistic.yaml --remove-meta"
('create_incremental_task_with_gtid_success resp=', {u'check_result': u'pre-check is passed. ', u'task': {u'source_config': {u'source_conf': [{u'source_name': u'mysql-01'}, {u'source_name': u'mysql-02'}]}, u'enhance_online_schema_change': True, u'on_duplicate': u'error', u'table_migrate_rule': [{u'source': {u'table': u'*', u'source_name': u'mysql-01', u'schema': u'openapi'}, u'target': {u'table': u'', u'schema': u'openapi'}}, {u'source': {u'table': u'*', u'source_name': u'mysql-02', u'schema': u'openapi'}, u'target': {u'table': u'', u'schema': u'openapi'}}], u'target_config': {u'user': u'root', u'host': u'127.0.0.1', u'password': u'', u'port': 4000, u'security': None}, u'meta_schema': u'dm_meta', u'task_mode': u'incremental', u'name': u'incremental_task_use_start_time_after_create'}})
wait process dm-master.test exit...
process dm-master.test already exit
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 12:35:01 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/extend_column/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
dmctl test cmd: "query-status incremental_task_use_start_time_after_create"
got=2 expected=2
check diff successfully
wait for port 8262 offline the 1-th time
dmctl test cmd: "config source mysql-replica-01"
got=1 expected=1
got=1 expected=1
wait process dm-master.test exit...
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 12:35:02 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/all_mode/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
dmctl test cmd: "query-status test"
check diff failed 2-th time, retry later
wait process dm-worker.test exit...
got=2 expected=2
run tidb sql failed 1-th time, retry later
wait for port 8262 offline the 2-th time
rpc addr 127.0.0.1:8262 is alive
[Sat May 4 12:35:03 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/extend_column/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
[Sat May 4 12:35:03 CST 2024] <<<<<< finish DM-4185 optimistic >>>>>>
rpc addr 127.0.0.1:8262 is alive
[Sat May 4 12:35:03 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/all_mode/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
dmctl test cmd: "query-status test"
wait process dm-master.test exit...
[Sat May 4 12:35:04 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/sequence_safe_mode/conf/dm-worker1.toml >>>>>>
[Sat May 4 12:35:04 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/sequence_safe_mode/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
got=1 expected=1
check diff successfully
dmctl test cmd: "validation status test"
got=1 expected=1
dmctl test cmd: "validation start test"
wait process dm-worker.test exit...
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/extend_column/source1.yaml"
dmctl test cmd: "operate-source create /tmp/dm_test/extend_column/source2.yaml"
dmctl test cmd: "validation status test"
got=1 expected=1
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/all_mode/source1.yaml"
check diff failed 3-th time, retry later
[Sat May 4 12:35:04 CST 2024] <<<<<< start DM-4201 >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/handle_error_2/conf/single-source-no-sharding.yaml --remove-meta"
wait process dm-master.test exit...
dmctl test cmd: "operate-source create /tmp/dm_test/all_mode/source2.yaml"
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 12:35:04 CST 2024] <<<<<< test case foreign_key success! >>>>>>
start running case: [full_mode] script: [/home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/full_mode/run.sh]
Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/full_mode/run.sh...
Verbose mode = false
0 dm-master alive
0 dm-worker alive
0 dm-syncer alive
process dm-master.test already exit
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 12:35:05 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/full_mode/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
rpc addr 127.0.0.1:8262 is alive
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "stop-task test"
[Sat May 4 12:35:05 CST 2024] <<<<<< finish DM-106 optimistic >>>>>>
dmctl test cmd: "start-task /tmp/dm_test/extend_column/dm-task.yaml --remove-meta"
wait process dm-master.test exit...
dmctl test cmd: "start-task /tmp/dm_test/extend_column/dm-task.yaml --remove-meta"
dmctl test cmd: "query-status test"
got=1 expected=1
dmctl test cmd: "query-status test"
start_task_with_condition success
dmctl test cmd: "query-status incremental_task_use_start_time_after_create"
got=2 expected=2
dmctl test cmd: "validation status test"
got=0 expected=1
command: validation status test "processedRowsStatus": "insert\/update\/delete: 2\/1\/1" count: 0 != expected: 1, failed the 0-th time, will retry again
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/all_mode/conf/dm-task-expression-filter.yaml --remove-meta"
[Sat May 4 12:35:06 CST 2024] <<<<<< start DM-107 optimistic >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl3_1/conf/double-source-optimistic.yaml --remove-meta"
got=1 expected=1
dmctl test cmd: "binlog skip test -b dm-it-175e207b-2603-4dff-98a7-3fe52f932268-n5x6k-7hgzv-bin|000001.000001:30234"
wait for rpc addr 127.0.0.1:8261 alive the 2-th time
check diff failed at last
check diff successfully
dmctl test cmd: "stop-task test"
dmctl test cmd: "query-status test"
wait process dm-master.test exit...
delete_task_success
('get_task_list resp=', {u'total': 0, u'data': []})
got=1 expected=1
dmctl test cmd: "binlog skip test"
dmctl test cmd: "query-status test"
('create_incremental_task_with_gtid_success resp=', {u'check_result': u'pre-check is passed. ', u'task': {u'source_config': {u'source_conf': [{u'source_name': u'mysql-01', u'binlog_gtid': u'bb287b4c-09ce-11ef-a980-d61658047e35:1-70', u'binlog_name': u'dm-it-367ac2a8-0da3-4b83-82ef-26d760a69262-zf74z-8p4tb-bin.000001', u'binlog_pos': 13696}, {u'source_name': u'mysql-02', u'binlog_gtid': u'bb91b73d-09ce-11ef-950b-d61658047e35:1-66', u'binlog_name': u'dm-it-367ac2a8-0da3-4b83-82ef-26d760a69262-zf74z-8p4tb-bin.000001', u'binlog_pos': 14442}]}, u'enhance_online_schema_change': True, u'on_duplicate': u'error', u'table_migrate_rule': [{u'source': {u'table': u'*', u'source_name': u'mysql-01', u'schema': u'openapi'}, u'target': {u'table': u'', u'schema': u'openapi'}}, {u'source': {u'table': u'*', u'source_name': u'mysql-02', u'schema': u'openapi'}, u'target': {u'table': u'', u'schema': u'openapi'}}], u'target_config': {u'user': u'root', u'host': u'127.0.0.1', u'password': u'', u'port': 4000, u'security': None}, u'meta_schema': u'dm_meta', u'task_mode': u'incremental', u'name': u'incremental_task_both_gtid_start_time'}})
dmctl test cmd: "query-status incremental_task_both_gtid_start_time"
got=2 expected=2
check sync diff after reset failpoint
got=2 expected=2
check diff successfully
[Sat May 4 12:35:07 CST 2024] <<<<<< finish DM-003 pessimistic >>>>>>
dmctl test cmd: "stop-task test"
[Sat May 4 12:35:07 CST 2024] <<<<<< finish DM-4201 >>>>>>
run tidb sql failed 1-th time, retry later
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 12:35:07 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/full_mode/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
wait process dm-master.test exit...
check diff successfully
check diff failed 1-th time, retry later
dmctl test cmd: "query-status test"
got=2 expected=2
dmctl test cmd: "query-status test"
got=3 expected=3
wait process dm-worker.test exit...
[Sat May 4 12:35:08 CST 2024] <<<<<< start DM-004 optimistic >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl1/conf/single-source-optimistic.yaml --remove-meta"
got=1 expected=1
got=0 expected=1
command: validation status test pendingRowsStatus": "insert\/update\/delete: 0\/0\/0 count: 0 != expected: 1, failed the 1-th time, will retry again
wait process dm-master.test exit...
1 dm-master alive
2 dm-worker alive
0 dm-syncer alive
rpc addr 127.0.0.1:8262 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/full_mode/source1.yaml"
[Sat May 4 12:35:08 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/full_mode/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
wait process dm-worker.test exit...
process dm-worker.test already exit
restart DM-worker after set SequenceShardSyncedExecutionExit failpoint
[Sat May 4 12:35:09 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/sequence_safe_mode/conf/dm-worker1.toml >>>>>>
[Sat May 4 12:35:09 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/sequence_safe_mode/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
wait process dm-master.test exit...
dmctl test cmd: "query-status test"
got=1 expected=1
run tidb sql failed 1-th time, retry later
check diff failed 2-th time, retry later
1 dm-master alive
2 dm-worker alive
0 dm-syncer alive
wait process dm-master.test exit...
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/full_mode/source2.yaml"
rpc addr 127.0.0.1:8262 is alive
rpc addr 127.0.0.1:8263 is alive
wait process dm-master.test exit...
wait process dm-master.test exit...
process dm-master.test already exit
got=1 expected=1
got=1 expected=1
got=1 expected=1
check diff successfully
--> start validator on the fly, validate from current syncer progress(gtid=false, relay=true)
wait process dm-master.test exit...
dmctl test cmd: "start-task /tmp/dm_test/full_mode/dm-task.yaml --remove-meta"
start_task_with_condition success
dmctl test cmd: "query-status incremental_task_both_gtid_start_time"
wait process dm-master.test exit...
process dm-master.test already exit
check diff successfully
dmctl test cmd: "stop-task test"
wait process dm-worker.test exit...
check diff successfully
dmctl test cmd: "stop-task test"
got=2 expected=2
[Sat May 4 12:35:11 CST 2024] <<<<<< finish DM-107 optimistic >>>>>>
1 dm-master alive
2 dm-worker alive
0 dm-syncer alive
delete_task_success
('get_task_list resp=', {u'total': 0, u'data': []})
wait process dm-master.test exit...
[Sat May 4 12:35:11 CST 2024] <<<<<< finish DM-004 optimistic >>>>>>
check diff failed 3-th time, retry later
wait process dm-worker.test exit...
[Sat May 4 12:35:12 CST 2024] <<<<<< start DM-108 optimistic >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl3_1/conf/double-source-optimistic.yaml --remove-meta"
wait process dm-master.test exit...
wait process dm-worker.test exit...
wait process dm-master.test exit...
wait for one of DM-worker offine failed, retry later
[Sat May 4 12:35:12 CST 2024] <<<<<< start DM-005 >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl1/conf/single-source-pessimistic.yaml --remove-meta"
wait process dm-worker.test exit...
dmctl test cmd: "query-status test"
got=2 expected=2
got=1 expected=1
got=1 expected=1
1 dm-master alive
2 dm-worker alive
0 dm-syncer alive
wait process dm-master.test exit...
process dm-master.test already exit
wait process dm-worker.test exit...
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 12:35:13 CST 2024] <<<<<< test case handle_error_2 success! >>>>>>
start running case: [handle_error_3] script: [/home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/handle_error_3/run.sh]
Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/handle_error_3/run.sh...
Verbose mode = false
0 dm-master alive
0 dm-worker alive
0 dm-syncer alive
process dm-master.test already exit
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 12:35:13 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/handle_error_3/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
wait process dm-master.test exit...
wait for one of DM-worker offine failed, retry later
dmctl test cmd: "query-status test"
got=2 expected=2
dmctl test cmd: "query-status test"
check diff failed at last
dmctl test cmd: "query-status test"
1 dm-master alive
2 dm-worker alive
0 dm-syncer alive
wait process dm-worker.test exit...
process dm-worker.test already exit
[Sat May 4 12:35:13 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/openapi/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
got=1 expected=1
got=1 expected=1
check diff failed 1-th time, retry later
dmctl test cmd: "query-status test"
got=1 expected=1
wait process dm-master.test exit...
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
running extend_column case with import_mode: loader
[Sat May 4 12:35:14 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/extend_column/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
check diff successfully
dmctl test cmd: "stop-task test"
wait process dm-master.test exit...
DM-worker1 is sharding lock owner and detects it offline
[Sat May 4 12:35:14 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/sequence_safe_mode/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
[Sat May 4 12:35:14 CST 2024] <<<<<< finish DM-005 >>>>>>
wait process dm-master.test exit...
wait process dm-worker.test exit...
rpc addr 127.0.0.1:8262 is alive
[Sat May 4 12:35:15 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/openapi/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
wait for rpc addr 127.0.0.1:8261 alive the 2-th time
wait process dm-master.test exit...
process dm-master.test already exit
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 12:35:15 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/extend_column/conf/dm-worker1.toml >>>>>>
rpc addr 127.0.0.1:8262 is alive
1 dm-master alive
2 dm-worker alive
0 dm-syncer alive
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
wait process dm-master.test exit...
process dm-master.test already exit
wait process dm-worker.test exit...
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 12:35:15 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/handle_error_3/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
wait process dm-master.test exit...
rpc addr 127.0.0.1:8263 is alive
('list_source_by_openapi_success resp=', {u'total': 2, u'data': [{u'enable_gtid': False, u'enable': True, u'source_name': u'mysql-01', u'security': None, u'relay_config': {u'relay_binlog_name': u'', u'relay_dir': u'relay-dir', u'relay_binlog_gtid': u'', u'enable_relay': False}, u'purge': {u'remain_space': 15, u'expires': 0, u'interval': 3600}, u'host': u'127.0.0.1', u'user': u'root', u'flavor': u'mysql', u'password': u'******', u'port': 3306}, {u'enable_gtid': False, u'enable': True, u'source_name': u'mysql-02', u'security': None, u'relay_config': {u'relay_binlog_name': u'', u'relay_dir': u'relay-dir', u'relay_binlog_gtid': u'', u'enable_relay': False}, u'purge': {u'remain_space': 15, u'expires': 0, u'interval': 3600}, u'host': u'127.0.0.1', u'user': u'root', u'flavor': u'mysql', u'password': u'******', u'port': 3307}]})
check diff failed 2-th time, retry later
rpc addr 127.0.0.1:8262 is alive
[Sat May 4 12:35:16 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/extend_column/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
wait process dm-worker.test exit...
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 12:35:16 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/validator_basic/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
rpc addr 127.0.0.1:8262 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/handle_error_3/source1.yaml"
wait process dm-master.test exit...
[Sat May 4 12:35:17 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/handle_error_3/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
wait process dm-master.test exit...
wait process dm-worker.test exit...
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/extend_column/source1.yaml"
wait process dm-worker.test exit...
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 12:35:17 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/validator_basic/conf/dm-worker1.toml >>>>>>
wait process dm-master.test exit...
dmctl test cmd: "operate-source create /tmp/dm_test/extend_column/source2.yaml"
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
wait process dm-master.test exit...
process dm-master.test already exit
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/handle_error_3/source2.yaml"
wait process dm-worker.test exit...
check diff failed 3-th time, retry later
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 12:35:18 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/full_mode/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
wait process dm-master.test exit...
rpc addr 127.0.0.1:8262 is alive
dmctl test cmd: "operate-source create /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/validator_basic/conf/source1.yaml"
wait process dm-worker.test exit...
[Sat May 4 12:35:19 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/validator_basic/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
dmctl test cmd: "start-task /tmp/dm_test/extend_column/dm-task.yaml --remove-meta"
[Sat May 4 12:35:19 CST 2024] <<<<<< start DM-4189 pessimistic >>>>>>
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 12:35:19 CST 2024] <<<<<< finish test_expression_filter >>>>>>
[Sat May 4 12:35:19 CST 2024] <<<<<< start test_json_expression >>>>>>
[Sat May 4 12:35:19 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/all_mode/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
dmctl test cmd: "start-task /tmp/dm_test/extend_column/dm-task.yaml --remove-meta"
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/handle_error_3/conf/double-source-pessimistic.yaml --remove-meta"
wait process dm-master.test exit...
wait process dm-worker.test exit...
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 12:35:19 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/full_mode/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 12:35:20 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/all_mode/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
check diff failed at last
dmctl test cmd: "stop-task test"
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/validator_basic/conf/dm-task-standalone-no-validator.yaml --remove-meta"
('create_incremental_task_with_gtid_success resp=', {u'check_result': u'pre-check is passed. ', u'task': {u'source_config': {u'source_conf': [{u'source_name': u'mysql-01'}, {u'source_name': u'mysql-02'}]}, u'enhance_online_schema_change': True, u'on_duplicate': u'error', u'table_migrate_rule': [{u'source': {u'table': u'*', u'source_name': u'mysql-01', u'schema': u'openapi'}, u'target': {u'table': u'', u'schema': u'openapi'}}, {u'source': {u'table': u'*', u'source_name': u'mysql-02', u'schema': u'openapi'}, u'target': {u'table': u'', u'schema': u'openapi'}}], u'target_config': {u'user': u'root', u'host': u'127.0.0.1', u'password': u'', u'port': 4000, u'security': None}, u'meta_schema': u'dm_meta', u'task_mode': u'incremental', u'name': u'incremental_task_no_duration_but_error'}})
dmctl test cmd: "query-status incremental_task_no_duration_but_error"
wait process dm-master.test exit...
[Sat May 4 12:35:20 CST 2024] <<<<<< finish DM-108 optimistic >>>>>>
check sync diff after restart DDL owner
check diff successfully
1 dm-master alive
2 dm-worker alive
0 dm-syncer alive
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 12:35:20 CST 2024] <<<<<< test case shardddl1 success! >>>>>>
start running case: [shardddl1_1] script: [/home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl1_1/run.sh]
Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl1_1/run.sh...
Verbose mode = false
rpc addr 127.0.0.1:8262 is alive
[Sat May 4 12:35:20 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/full_mode/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
dmctl test cmd: "query-status test"
0 dm-master alive
0 dm-worker alive
0 dm-syncer alive
process dm-master.test already exit
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 12:35:21 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl1_1/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
got=2 expected=2
got=2 expected=2
start_task_with_condition success
dmctl test cmd: "query-status incremental_task_no_duration_but_error"
got=1 expected=2
command: query-status incremental_task_no_duration_but_error Duplicate entry count: 1 != expected: 2, failed the 0-th time, will retry again
dmctl test cmd: "config source mysql-replica-01"
1 dm-master alive
2 dm-worker alive
0 dm-syncer alive
[Sat May 4 12:35:21 CST 2024] <<<<<< start DM-109 optimistic >>>>>>
got=1 expected=1
got=1 expected=1
wait process dm-master.test exit...
rpc addr 127.0.0.1:8262 is alive
[Sat May 4 12:35:21 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/all_mode/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/full_mode/source1.yaml"
dmctl test cmd: "query-status test"
got=2 expected=2
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl3_1/conf/double-source-optimistic.yaml --remove-meta"
wait process dm-master.test exit...
wait process dm-master.test exit...
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 12:35:22 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl1_1/conf/dm-worker1.toml >>>>>>
dmctl test cmd: "operate-source create /tmp/dm_test/full_mode/source2.yaml"
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
dmctl test cmd: "binlog replace test -b dm-it-175e207b-2603-4dff-98a7-3fe52f932268-n5x6k-7hgzv-bin|000001.000001:1559 alter table handle_error.tb1 add column d varchar(20);"
dmctl test cmd: "query-status test"
wait process dm-master.test exit...
process dm-master.test already exit
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/all_mode/source1.yaml"
got=2 expected=2
dmctl test cmd: "binlog replace test -b dm-it-175e207b-2603-4dff-98a7-3fe52f932268-n5x6k-7hgzv-bin|000001.000001:1735 alter table handle_error.tb1 add column d varchar(20);"
dmctl test cmd: "operate-source create /tmp/dm_test/all_mode/source2.yaml"
wait process dm-master.test exit...
dmctl test cmd: "query-status test"
got=2 expected=2
dmctl test cmd: "binlog replace test alter table handle_error.tb1 add column c varchar(20);"
dmctl test cmd: "query-status test"
rpc addr 127.0.0.1:8262 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/shardddl1_1/source1.yaml"
dmctl test cmd: "query-status test"
dmctl test cmd: "query-status test"
got=2 expected=2
dmctl test cmd: "query-status test"
got=2 expected=2
got=1 expected=1
got=1 expected=1
check diff successfully
dmctl test cmd: "validation status test"
wait process dm-master.test exit...
process dm-master.test already exit
[Sat May 4 12:35:23 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl1_1/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
got=3 expected=3
dmctl test cmd: "stop-task test"
check diff failed 1-th time, retry later
got=1 expected=1
dmctl test cmd: "validation start test"
dmctl test cmd: "validation status test"
got=1 expected=1
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/all_mode/conf/dm-task-expression-filter.yaml --remove-meta"
wait process dm-master.test exit...
got=2 expected=2
start_task_with_condition success
run tidb sql failed 1-th time, retry later
wait process dm-worker.test exit...
[Sat May 4 12:35:24 CST 2024] <<<<<< finish DM-4189 pessimistic >>>>>>
[Sat May 4 12:35:24 CST 2024] <<<<<< start DM-4189 optimistic >>>>>>
wait process dm-worker.test exit...
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/handle_error_3/conf/double-source-optimistic.yaml --remove-meta"
wait process dm-master.test exit...
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/shardddl1_1/source2.yaml"
wait process dm-worker.test exit...
wait process dm-worker.test exit...
dmctl test cmd: "start-task /tmp/dm_test/full_mode/dm-task.yaml --remove-meta"
dmctl test cmd: "validation status test"
got=0 expected=1
command: validation status test "processedRowsStatus": "insert\/update\/delete: 2\/1\/1" count: 0 != expected: 1, failed the 0-th time, will retry again
check diff failed 1-th time, retry later
wait process dm-master.test exit...
[Sat May 4 12:35:25 CST 2024] <<<<<< start DM-SAME_DDL_TWICE pessimistic >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl1_1/conf/double-source-pessimistic.yaml --remove-meta"
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 12:35:25 CST 2024] <<<<<< test case sequence_safe_mode success! >>>>>>
start running case: [lightning_load_task] script: [/home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/lightning_load_task/run.sh]
Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/lightning_load_task/run.sh...
Verbose mode = false
0 dm-master alive
0 dm-worker alive
0 dm-syncer alive
process dm-master.test already exit
process dm-worker.test already exit
process dm-syncer.test already exit
import prepare data
start DM master, workers and sources
[Sat May 4 12:35:25 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/lightning_load_task/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
dmctl test cmd: "query-status test"
got=2 expected=2
check diff failed 2-th time, retry later
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 12:35:26 CST 2024] <<<<<< test case extend_column success! >>>>>>
start running case: [fake_rotate_event] script: [/home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/fake_rotate_event/run.sh]
Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/fake_rotate_event/run.sh...
Verbose mode = false
0 dm-master alive
0 dm-worker alive
0 dm-syncer alive
process dm-master.test already exit
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 12:35:26 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/fake_rotate_event/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
check diff failed 1-th time, retry later
delete_task_success
('get_task_list resp=', {u'total': 0, u'data': []})
delete_source_with_force_success
delete_source_with_force_success
('list_source_by_openapi_success resp=', {u'total': 0, u'data': []})
('get_task_list resp=', {u'total': 0, u'data': []})
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>TEST OPENAPI: START TASK WITH CONDITION SUCCESS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>START TEST OPENAPI: STOP TASK WITH CONDITION
wait process dm-master.test exit...
('create_source1_success resp=', {u'enable_gtid': False, u'enable': True, u'source_name': u'mysql-01', u'host': u'127.0.0.1', u'user': u'root', u'security': None, u'password': u'123456', u'port': 3306})
('list_source_by_openapi_success resp=', {u'total': 1, u'data': [{u'enable_gtid': False, u'enable': True, u'source_name': u'mysql-01', u'security': None, u'relay_config': {u'relay_binlog_name': u'', u'relay_dir': u'relay-dir', u'relay_binlog_gtid': u'', u'enable_relay': False}, u'purge': {u'remain_space': 15, u'expires': 0, u'interval': 3600}, u'host': u'127.0.0.1', u'user': u'root', u'flavor': u'mysql', u'password': u'******', u'port': 3306}]})
dmctl test cmd: "query-status test"
got=2 expected=2
('get_source_status_success resp=', {u'total': 1, u'data': [{u'worker_name': u'worker2', u'source_name': u'mysql-01'}]})
('create_source1_success resp=', {u'enable_gtid': False, u'enable': True, u'source_name': u'mysql-02', u'host': u'127.0.0.1', u'user': u'root', u'security': None, u'password': u'123456', u'port': 3307})
('list_source_by_openapi_success resp=', {u'total': 2, u'data': [{u'enable_gtid': False, u'enable': True, u'source_name': u'mysql-01', u'security': None, u'relay_config': {u'relay_binlog_name': u'', u'relay_dir': u'relay-dir', u'relay_binlog_gtid': u'', u'enable_relay': False}, u'purge': {u'remain_space': 15, u'expires': 0, u'interval': 3600}, u'host': u'127.0.0.1', u'user': u'root', u'flavor': u'mysql', u'password': u'******', u'port': 3306}, {u'enable_gtid': False, u'enable': True, u'source_name': u'mysql-02', u'security': None, u'relay_config': {u'relay_binlog_name': u'', u'relay_dir': u'relay-dir', u'relay_binlog_gtid': u'', u'enable_relay': False}, u'purge': {u'remain_space': 15, u'expires': 0, u'interval': 3600}, u'host': u'127.0.0.1', u'user': u'root', u'flavor': u'mysql', u'password': u'******', u'port': 3307}]})
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 12:35:26 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/lightning_load_task/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
('get_source_status_success resp=', {u'total': 1, u'data': [{u'worker_name': u'worker1', u'source_name': u'mysql-02'}]})
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 12:35:27 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/fake_rotate_event/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
got=1 expected=1
got=0 expected=1
command: validation status test pendingRowsStatus": "insert\/update\/delete: 0\/0\/0 count: 0 != expected: 1, failed the 1-th time, will retry again
check diff successfully
1 dm-master alive
2 dm-worker alive
0 dm-syncer alive
wait process dm-master.test exit...
dmctl test cmd: "query-status test"
got=2 expected=2
dmctl test cmd: "binlog replace test -b dm-it-175e207b-2603-4dff-98a7-3fe52f932268-n5x6k-7hgzv-bin|000001.000001:3856 alter table handle_error.tb1 add column d varchar(20);"
check diff failed 3-th time, retry later
check diff failed 1-th time, retry later
dmctl test cmd: "query-status test"
rpc addr 127.0.0.1:8262 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/lightning_load_task/source1.yaml"
check diff successfully
check dump files have been cleaned
ls: cannot access /tmp/dm_test/full_mode/worker1/dumped_data.test: No such file or directory
worker1 auto removed dump files
ls: cannot access /tmp/dm_test/full_mode/worker2/dumped_data.test: No such file or directory
worker2 auto removed dump files
1 dm-master alive
2 dm-worker alive
0 dm-syncer alive
got=2 expected=2
dmctl test cmd: "binlog replace test -b dm-it-175e207b-2603-4dff-98a7-3fe52f932268-n5x6k-7hgzv-bin|000001.000001:4247 alter table handle_error.tb1 add column d varchar(20);"
wait process dm-worker.test exit...
[Sat May 4 12:35:28 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/lightning_load_task/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
rpc addr 127.0.0.1:8262 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/fake_rotate_event/source1.yaml"
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/fake_rotate_event/conf/dm-task.yaml --remove-meta"
wait process dm-master.test exit...
dmctl test cmd: "query-status test"
wait process dm-master.test exit...
wait process dm-master.test exit...
got=2 expected=2
dmctl test cmd: "binlog replace test alter table handle_error.tb1 add column c varchar(20);"
wait process dm-worker.test exit...
process dm-worker.test already exit
[Sat May 4 12:35:29 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/openapi/conf/dm-worker1.toml >>>>>>
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/lightning_load_task/source2.yaml"
dmctl test cmd: "query-status test"
wait process dm-master.test exit...
process dm-master.test already exit
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
got=3 expected=3
dmctl test cmd: "query-status test"
got=2 expected=2
got=1 expected=1
check diff failed at last
dmctl test cmd: "stop-task test"
check diff successfully
dmctl test cmd: "stop-task test"
run tidb sql failed 1-th time, retry later
wait process dm-master.test exit...
check diff successfully
kill dm-worker
got=1 expected=1
got=1 expected=1
got=1 expected=1
check diff successfully
--> start validator on the fly, validate from current syncer progress(gtid=true, relay=false)
wait process dm-master.test exit...
[Sat May 4 12:35:29 CST 2024] <<<<<< finish DM-109 optimistic >>>>>>
process dm-master.test already exit
wait process dm-worker.test exit...
[Sat May 4 12:35:30 CST 2024] <<<<<< finish DM-SAME_DDL_TWICE pessimistic >>>>>>
[Sat May 4 12:35:30 CST 2024] <<<<<< start DM-BINARY_COLUMN >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl1_1/conf/single-source-no-sharding.yaml --remove-meta"
rpc addr 127.0.0.1:8262 is alive
[Sat May 4 12:35:30 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/openapi/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
[Sat May 4 12:35:30 CST 2024] <<<<<< START DM-WORKER on port 8264, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/lightning_load_task/conf/dm-worker3.toml >>>>>>
wait for rpc addr 127.0.0.1:8264 alive the 1-th time
wait process dm-master.test exit...
wait process dm-worker1 exit...
1 dm-master alive
2 dm-worker alive
0 dm-syncer alive
[Sat May 4 12:35:30 CST 2024] <<<<<< start DM-110 optimistic >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl3_1/conf/double-source-optimistic.yaml --remove-meta"
wait process dm-worker.test exit...
rpc addr 127.0.0.1:8264 is alive
start DM task
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/lightning_load_task/conf/dm-task.yaml --remove-meta"
wait process dm-worker.test exit...
dmctl test cmd: "query-status test"
got=1 expected=1
check diff successfully
check diff successfully
wait process dm-master.test exit...
rpc addr 127.0.0.1:8263 is alive
('create_noshard_task_success resp=', {u'check_result': u'pre-check is passed. ', u'task': {u'source_config': {u'source_conf': [{u'source_name': u'mysql-01'}, {u'source_name': u'mysql-02'}]}, u'enhance_online_schema_change': True, u'on_duplicate': u'error', u'table_migrate_rule': [{u'source': {u'table': u'*', u'source_name': u'mysql-01', u'schema': u'openapi'}, u'target': {u'table': u'', u'schema': u'openapi'}}, {u'source': {u'table': u'*', u'source_name': u'mysql-02', u'schema': u'openapi'}, u'target': {u'table': u'', u'schema': u'openapi'}}], u'target_config': {u'user': u'root', u'host': u'127.0.0.1', u'password': u'', u'port': 4000, u'security': None}, u'meta_schema': u'dm-meta', u'task_mode': u'all', u'name': u'test_wait_time_on_stop'}})
dmctl test cmd: "query-status test_wait_time_on_stop"
got=2 expected=2
wait process dm-worker1 exit...
process dm-worker1 already exit
[Sat May 4 12:35:31 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/fake_rotate_event/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
wait process dm-master.test exit...
check diff successfully
check diff successfully
check diff successfully
dmctl test cmd: "query-status test_wait_time_on_stop"
got=2 expected=2
wait process dm-worker.test exit...
check diff successfully
dmctl test cmd: "stop-task test"
check diff successfully
error check
process dm-worker.test already exit
[Sat May 4 12:35:32 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/openapi/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
dmctl test cmd: "query-status test"
got=2 expected=2
dmctl test cmd: "stop-task test"
[Sat May 4 12:35:32 CST 2024] <<<<<< finish DM-BINARY_COLUMN >>>>>>
[Sat May 4 12:35:32 CST 2024] <<<<<< start DM-006 >>>>>>
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 12:35:32 CST 2024] <<<<<< finish test_json_expression >>>>>>
[Sat May 4 12:35:32 CST 2024] <<<<<< start test_fail_job_between_event >>>>>>
[Sat May 4 12:35:32 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/all_mode/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
dmctl test cmd: "query-status test"
got=2 expected=2
got=1 expected=1
check diff failed 1-th time, retry later
wait process dm-master.test exit...
wait process dm-master.test exit...
process dm-master.test already exit
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl1_1/conf/double-source-pessimistic.yaml --remove-meta"
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/lightning_load_task/conf/dm-task2.yaml --remove-meta"
wait process dm-worker.test exit...
rpc addr 127.0.0.1:8262 is alive
check diff successfully
dmctl test cmd: "query-status test"
got=2 expected=2
got=1 expected=1
1 dm-master alive
1 dm-worker alive
0 dm-syncer alive
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 12:35:33 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/full_mode/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
wait process dm-master.test exit...
rpc addr 127.0.0.1:8262 is alive
[Sat May 4 12:35:33 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/openapi/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
wait process dm-worker.test exit...
[Sat May 4 12:35:33 CST 2024] <<<<<< finish DM-4189 optimistic >>>>>>
[Sat May 4 12:35:33 CST 2024] <<<<<< start DM-4189 pessimistic >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 2-th time
wait process dm-master.test exit...
dmctl test cmd: "query-status test"
got=2 expected=2
check diff failed 1-th time, retry later
dmctl test cmd: "query-status load_task1"
got=1 expected=1
got=1 expected=1
dmctl test cmd: "query-status load_task2"
got=1 expected=1
got=1 expected=1
test worker restart
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/handle_error_3/conf/double-source-pessimistic.yaml --remove-meta"
rpc addr 127.0.0.1:8263 is alive
delete_task_success
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 12:35:34 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/full_mode/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 12:35:34 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/all_mode/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
('get_task_list resp=', {u'total': 0, u'data': []})
delete_source_with_force_success
wait process dm-master.test exit...
wait process dm-worker.test exit...
check diff failed 2-th time, retry later
delete_source_with_force_success
('list_source_by_openapi_success resp=', {u'total': 0, u'data': []})
wait process dm-worker1 exit...
('get_task_list resp=', {u'total': 0, u'data': []})
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>TEST OPENAPI: START TASK WITH CONDITION SUCCESS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>START TEST OPENAPI: REVERSE HTTPS
2 dm-master alive
2 dm-worker alive
0 dm-syncer alive
wait process dm-master.test exit...
process dm-master.test already exit
rpc addr 127.0.0.1:8262 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/full_mode/source1.yaml"
[Sat May 4 12:35:35 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/full_mode/conf/dm-worker2.toml >>>>>>
dmctl test cmd: "query-status test"
rpc addr 127.0.0.1:8262 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/all_mode/source1.yaml"
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
got=2 expected=2
wait process dm-master.test exit...
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 12:35:35 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/validator_basic/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
wait process dm-master.test exit...
wait process dm-worker.test exit...
wait process dm-worker1 exit...
process dm-worker1 already exit
dmctl test cmd: "list-member -w -n worker3"
got=1 expected=1
got=1 expected=1
dmctl test cmd: "list-member -w -n worker1"
got=1 expected=1
dmctl test cmd: "query-status load_task1"
check diff successfully
dmctl test cmd: "stop-task test"
[Sat May 4 12:35:36 CST 2024] <<<<<< finish DM-006 >>>>>>
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/full_mode/source2.yaml"
got=1 expected=1
got=1 expected=1
[Sat May 4 12:35:36 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/lightning_load_task/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
wait process dm-master.test exit...
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 12:35:37 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/validator_basic/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
check diff failed 3-th time, retry later
dmctl test cmd: "query-status test"
got=2 expected=2
wait process dm-worker.test exit...
dmctl test cmd: "binlog replace test -b dm-it-175e207b-2603-4dff-98a7-3fe52f932268-n5x6k-7hgzv-bin|000001.000001:6411 alter table handle_error.tb1 add column d varchar(20);"
[Sat May 4 12:35:37 CST 2024] <<<<<< start DM-007 >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl1_1/conf/double-source-pessimistic.yaml --remove-meta"
[Sat May 4 12:35:37 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/all_mode/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
wait process dm-master.test exit...
process dm-master.test already exit
rpc addr 127.0.0.1:8262 is alive
dmctl test cmd: "list-member -w -n worker3"
dmctl test cmd: "query-status test"
got=2 expected=2
dmctl test cmd: "binlog replace test -b dm-it-175e207b-2603-4dff-98a7-3fe52f932268-n5x6k-7hgzv-bin|000001.000001:6759 alter table handle_error.tb1 add column d varchar(20);"
wait process dm-master.test exit...
got=1 expected=1
dmctl test cmd: "list-member -w -n worker1"
got=1 expected=1
got=1 expected=1
dmctl test cmd: "query-status load_task1"
dmctl test cmd: "query-status test"
got=2 expected=2
dmctl test cmd: "binlog replace test alter table handle_error.tb1 add column c varchar(20);"
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/full_mode/conf/dm-task.yaml --remove-meta"
got=1 expected=1
got=1 expected=1
dmctl test cmd: "query-status load_task2"
got=1 expected=1
got=1 expected=1
test_transfer_two_sources
rpc addr 127.0.0.1:8262 is alive
dmctl test cmd: "operate-source create /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/validator_basic/conf/source1.yaml"
[Sat May 4 12:35:38 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/validator_basic/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 12:35:38 CST 2024] <<<<<< test case fake_rotate_event success! >>>>>>
dmctl test cmd: "query-status test"
[Pipeline] }
Cache not saved (ws/jenkins-pingcap-tiflow-pull_dm_integration_test-1860/tiflow-dm already exists)
[Pipeline] // cache
[Pipeline] }
[Pipeline] // dir
got=3 expected=3
dmctl test cmd: "stop-task test"
[Pipeline] }
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/all_mode/source2.yaml"
wait process dm-worker.test exit...
[Pipeline] // withCredentials
[Pipeline] }
wait process dm-master.test exit...
[Pipeline] // timeout
[Sat May 4 12:35:38 CST 2024] <<<<<< finish DM-4189 pessimistic >>>>>>
[Sat May 4 12:35:38 CST 2024] <<<<<< start DM-4189 optimistic >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/handle_error_3/conf/double-source-optimistic.yaml --remove-meta"
[Pipeline] }
check diff failed at last
dmctl test cmd: "stop-task test"
[Pipeline] // stage
dmctl test cmd: "query-status test"
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
got=2 expected=2
check diff failed 1-th time, retry later
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] }
check diff successfully
[Sat May 4 12:35:39 CST 2024] <<<<<< finish DM-110 optimistic >>>>>>
[Pipeline] // withEnv
wait process dm-worker.test exit...
[Pipeline] }
wait process dm-worker2 exit...
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/validator_basic/conf/dm-task-standalone-no-validator.yaml --remove-meta"
[Pipeline] // stage
wait process dm-master.test exit...
[Pipeline] }
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/all_mode/conf/dm-task.yaml --remove-meta"
dmctl test cmd: "query-status test"
got=2 expected=2
dmctl test cmd: "query-status test"
1 dm-master alive
2 dm-worker alive
0 dm-syncer alive
[Sat May 4 12:35:40 CST 2024] <<<<<< start DM-111 optimistic >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl3_1/conf/double-source-optimistic.yaml --remove-meta"
dmctl test cmd: "query-status test"
got=2 expected=2
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 12:35:40 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /tmp/dm_test/openapi/dm-master1.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
wait process dm-worker2 exit...
process dm-worker2 already exit
dmctl test cmd: "list-member -w -n worker3"
got=1 expected=1
got=1 expected=1
dmctl test cmd: "query-status load_task2"
got=1 expected=1
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/lightning_load_task/conf/dm-task3.yaml --remove-meta"
dmctl test cmd: "config source mysql-replica-01"
got=1 expected=1
got=1 expected=1
dmctl test cmd: "binlog replace test -b dm-it-175e207b-2603-4dff-98a7-3fe52f932268-n5x6k-7hgzv-bin|000001.000001:8966 alter table handle_error.tb1 add column d varchar(20);"
dmctl test cmd: "query-status test"
got=2 expected=2
dmctl test cmd: "binlog replace test -b dm-it-175e207b-2603-4dff-98a7-3fe52f932268-n5x6k-7hgzv-bin|000001.000001:9271 alter table handle_error.tb1 add column d varchar(20);"
wait process dm-master.test exit...
dmctl test cmd: "query-status test"
check log contain failed 1-th time, retry later
wait process dm-master.test exit...
got=2 expected=2
dmctl test cmd: "binlog replace test alter table handle_error.tb1 add column c varchar(20);"
dmctl test cmd: "query-status test"
got=3 expected=3
dmctl test cmd: "stop-task test"
check diff successfully
dmctl test cmd: "stop-task test"
[Sat May 4 12:35:41 CST 2024] <<<<<< finish DM-007 >>>>>>
dmctl test cmd: "query-status test"
got=2 expected=2
dmctl test cmd: "query-status test"
wait for rpc addr 127.0.0.1:8261 alive the 2-th time
got=2 expected=2
dmctl test cmd: "query-status load_task3"
got=1 expected=1
[Sat May 4 12:35:41 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/lightning_load_task/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
[Sat May 4 12:35:41 CST 2024] <<<<<< finish DM-4189 optimistic >>>>>>
wait process dm-master.test exit...
got=1 expected=1
got=1 expected=1
check diff failed 1-th time, retry later
dmctl test cmd: "query-status test"
got=1 expected=1
check diff successfully
dmctl test cmd: "validation status test"
wait process dm-master.test exit...
process dm-master.test already exit
[Sat May 4 12:35:42 CST 2024] <<<<<< start DM-008 >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl1_1/conf/double-source-pessimistic.yaml --remove-meta"
got=1 expected=1
dmctl test cmd: "validation start test"
dmctl test cmd: "validation status test"
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 12:35:42 CST 2024] <<<<<< START DM-MASTER on port 8361, config: /tmp/dm_test/openapi/dm-master2.toml >>>>>>
wait for rpc addr 127.0.0.1:8361 alive the 1-th time
[Sat May 4 12:35:42 CST 2024] <<<<<< start DM-4210 >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/handle_error_3/conf/single-source-no-sharding.yaml --remove-meta"
wait process dm-master.test exit...
got=1 expected=1
dmctl test cmd: "query-status test"
got=3 expected=3
check diff successfully
1 dm-master alive
2 dm-worker alive
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "list-member -w -n worker2"
got=1 expected=1
0 dm-syncer alive
wait process dm-worker.test exit...
wait process dm-master.test exit...
dmctl test cmd: "query-status test"
got=2 expected=2
wait process dm-worker1 exit...
dmctl test cmd: "query-status test"
got=1 expected=1
check diff failed 1-th time, retry later
wait process dm-master.test exit...
wait process dm-worker.test exit...
dmctl test cmd: "query-status test"
got=1 expected=1
check diff failed 2-th time, retry later
dmctl test cmd: "binlog replace test -b dm-it-175e207b-2603-4dff-98a7-3fe52f932268-n5x6k-7hgzv-bin|000001.000001:11239 alter table handle_error.tb1 add column d int;alter table handle_error.tb1 add unique(d);"
dmctl test cmd: "query-status test"
wait process dm-master.test exit...
dmctl test cmd: "validation status test"
got=0 expected=1
command: validation status test "processedRowsStatus": "insert\/update\/delete: 2\/1\/1" count: 0 != expected: 1, failed the 0-th time, will retry again
got=1 expected=1
dmctl test cmd: "binlog replace test alter table handle_error.tb1 add column e int unique;"
wait process dm-worker1 exit...
process dm-worker1 already exit
dmctl test cmd: "list-member -w -n worker2"
wait process dm-master.test exit...
process dm-master.test already exit
dmctl test cmd: "query-status test"
got=1 expected=1
dmctl test cmd: "binlog replace test alter table handle_error.tb1 add column c int;alter table handle_error.tb1 add unique(c);"
got=1 expected=1
got=1 expected=1
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/lightning_load_task/conf/dm-task4.yaml --remove-meta"
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 12:35:45 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/full_mode/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
dmctl test cmd: "query-status test"
got=2 expected=2
run tidb sql failed 1-th time, retry later
wait process dm-master.test exit...
check diff failed 3-th time, retry later
wait process dm-worker.test exit...
got=1 expected=1
got=0 expected=1
command: validation status test pendingRowsStatus": "insert\/update\/delete: 0\/0\/0 count: 0 != expected: 1, failed the 1-th time, will retry again
got=2 expected=2
dmctl test cmd: "query-status load_task4"
got=1 expected=1
[Sat May 4 12:35:46 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/lightning_load_task/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
check diff successfully
dmctl test cmd: "stop-task test"
[Sat May 4 12:35:46 CST 2024] <<<<<< finish DM-008 >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 2-th time
wait process dm-master.test exit...
wait process dm-worker.test exit...
rpc addr 127.0.0.1:8262 is alive
dmctl test cmd: "list-member -w -n worker1"
[Sat May 4 12:35:47 CST 2024] <<<<<< start DM-009 >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl1_1/conf/double-source-pessimistic.yaml --remove-meta"
rpc addr 127.0.0.1:8261 is alive
[Sat May 4 12:35:47 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/full_mode/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
got=1 expected=1
wait process dm-master.test exit...
dmctl test cmd: "stop-task test"
[Sat May 4 12:35:47 CST 2024] <<<<<< finish DM-4210 >>>>>>
check diff failed at last
dmctl test cmd: "stop-task test"
wait process dm-worker.test exit...
process dm-worker.test already exit
process dm-syncer.test already exit
[Sat May 4 12:35:48 CST 2024] <<<<<< finish test_fail_job_between_event >>>>>>
[Sat May 4 12:35:48 CST 2024] <<<<<< start test_session_config >>>>>>
[Sat May 4 12:35:48 CST 2024] <<<<<< START DM-MASTER on port 8261, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/all_mode/conf/dm-master.toml >>>>>>
wait for rpc addr 127.0.0.1:8261 alive the 1-th time
[Sat May 4 12:35:48 CST 2024] <<<<<< finish DM-111 optimistic >>>>>>
wait process dm-worker3 exit...
wait process dm-master.test exit...
[Sat May 4 12:35:48 CST 2024] <<<<<< start DM-4193 pessimistic >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/handle_error_3/conf/double-source-pessimistic.yaml --remove-meta"
dmctl test cmd: "query-status test"
got=2 expected=2
rpc addr 127.0.0.1:8262 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/full_mode/source1.yaml"
[Sat May 4 12:35:49 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/full_mode/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
got=1 expected=1
got=1 expected=1
got=1 expected=1
check diff successfully
--> start validator on the fly, validate from current syncer progress(gtid=true, relay=true)
check diff failed 1-th time, retry later
rpc addr 127.0.0.1:8261 is alive
wait process dm-worker3 exit...
process dm-worker3 already exit
[Sat May 4 12:35:49 CST 2024] <<<<<< START DM-WORKER on port 8262, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/all_mode/conf/dm-worker1.toml >>>>>>
wait for rpc addr 127.0.0.1:8262 alive the 1-th time
wait process dm-master.test exit...
[Sat May 4 12:35:49 CST 2024] <<<<<< start DM-112 optimistic >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl3_1/conf/double-source-optimistic.yaml --remove-meta"
dmctl test cmd: "list-member -w -n worker1"
got=1 expected=1
got=1 expected=1
dmctl test cmd: "query-status load_task1"
got=1 expected=1
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/full_mode/source2.yaml"
1 dm-master alive
2 dm-worker alive
0 dm-syncer alive
dmctl test cmd: "query-status load_task2"
got=1 expected=1
dmctl test cmd: "list-member -w -n worker1"
got=1 expected=1
got=0 expected=1
command: list-member -w -n worker1 "source": "mysql-replica-01" count: 0 != expected: 1, failed the 0-th time, will retry again
dmctl test cmd: "query-status test"
got=2 expected=2
dmctl test cmd: "query-status test"
got=2 expected=2
rpc addr 127.0.0.1:8262 is alive
[Sat May 4 12:35:50 CST 2024] <<<<<< START DM-WORKER on port 8263, config: /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/all_mode/conf/dm-worker2.toml >>>>>>
wait for rpc addr 127.0.0.1:8263 alive the 1-th time
wait process dm-master.test exit...
dmctl test cmd: "binlog skip test -s mysql-replica-01 -b dm-it-175e207b-2603-4dff-98a7-3fe52f932268-n5x6k-7hgzv-bin|000001.000001:13027"
dmctl test cmd: "query-status test"
wait process dm-master.test exit...
dmctl test cmd: "binlog skip test -s mysql-replica-02 -b dm-it-175e207b-2603-4dff-98a7-3fe52f932268-n5x6k-7hgzv-bin|000001.000001:12152"
dmctl test cmd: "query-status test"
got=2 expected=2
dmctl test cmd: "query-status test"
check diff successfully
dmctl test cmd: "stop-task test"
got=2 expected=2
dmctl test cmd: "binlog revert test -s mysql-replica-01 -b dm-it-175e207b-2603-4dff-98a7-3fe52f932268-n5x6k-7hgzv-bin|000001.000001:13027"
dmctl test cmd: "binlog revert test -s mysql-replica-02 -b dm-it-175e207b-2603-4dff-98a7-3fe52f932268-n5x6k-7hgzv-bin|000001.000001:12152"
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/full_mode/conf/dm-task-2.yaml --remove-meta"
got=1 expected=1
got=1 expected=1
check diff failed 1-th time, retry later
dmctl test cmd: "binlog skip test -s mysql-replica-01 -b dm-it-175e207b-2603-4dff-98a7-3fe52f932268-n5x6k-7hgzv-bin|000001.000001:13413"
dmctl test cmd: "binlog revert test -b dm-it-175e207b-2603-4dff-98a7-3fe52f932268-n5x6k-7hgzv-bin|000001.000001:13413"
wait process dm-master.test exit...
[Sat May 4 12:35:51 CST 2024] <<<<<< finish DM-009 >>>>>>
dmctl test cmd: "binlog skip test -s mysql-replica-01 -b dm-it-175e207b-2603-4dff-98a7-3fe52f932268-n5x6k-7hgzv-bin|000001.000001:13413"
dmctl test cmd: "binlog revert test -s mysql-replica-02 -b dm-it-175e207b-2603-4dff-98a7-3fe52f932268-n5x6k-7hgzv-bin|000001.000001:12590"
wait process dm-master.test exit...
process dm-master.test already exit
rpc addr 127.0.0.1:8263 is alive
dmctl test cmd: "operate-source create /tmp/dm_test/all_mode/source1.yaml"
dmctl test cmd: "operate-source create /tmp/dm_test/all_mode/source2.yaml"
dmctl test cmd: "query-status test"
got=2 expected=2
dmctl test cmd: "binlog skip test"
Aborted by Jenkins Admin
Sending interrupt signal to process
Killing processes
dmctl test cmd: "query-status test"
got=1 expected=1
dmctl test cmd: "binlog skip test"
kill finished with exit code 0
Sending interrupt signal to process
Killing processes
restore config
make: *** [dm_integration_test_in_group] Terminated
/home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/validator_basic/run.sh: line 53: 8937 Terminated wait_process_exit dm-worker.test
restore time_zone
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
script returned exit code 143
dmctl test cmd: "query-status test"
got=3 expected=3
dmctl test cmd: "stop-task test"
[Sat May 4 12:35:52 CST 2024] <<<<<< start DM-010 >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/shardddl1_1/conf/single-source-pessimistic.yaml --remove-meta"
kill finished with exit code 0
Sending interrupt signal to process
Killing processes
got=1 expected=1
got=0 expected=1
command: list-member -w -n worker1 "source": "mysql-replica-01" count: 0 != expected: 1, failed the 1-th time, will retry again
dmctl test cmd: "query-status test"
got=1 expected=1
1 dm-master alive
2 dm-worker alive
0 dm-syncer alive
wait for rpc addr 127.0.0.1:8361 alive the 2-th time
make: *** [dm_integration_test_in_group] Terminated
script returned exit code 143
kill finished with exit code 0
Sending interrupt signal to process
Killing processes
make: *** [dm_integration_test_in_group] Terminated
script returned exit code 143
wait process dm-master.test exit...
[Sat May 4 12:35:53 CST 2024] <<<<<< finish DM-4193 pessimistic >>>>>>
[Sat May 4 12:35:53 CST 2024] <<<<<< start DM-4193 optimistic >>>>>>
dmctl test cmd: "start-task /home/jenkins/agent/workspace/pingcap/tiflow/pull_dm_integration_test/tiflow/dm/tests/handle_error_3/conf/double-source-optimistic.yaml --remove-meta"
kill finished with exit code 0
Sending interrupt signal to process
Killing processes
make: *** [dm_integration_test_in_group] Terminated
script returned exit code 143
kill finished with exit code 0
Sending interrupt signal to process
Killing processes
make: *** [dm_integration_test_in_group] Terminated
script returned exit code 143
wait process dm-master.test exit...
wait process dm-master.test exit...
kill finished with exit code 0
Sending interrupt signal to process
Killing processes
kill finished with exit code 0
Sending interrupt signal to process
Killing processes
dmctl test cmd: "query-status test"
got=2 expected=2
dmctl test cmd: "query-status test"
got=2 expected=2
make: *** [dm_integration_test_in_group] Terminated
make: *** [dm_integration_test_in_group] Terminated
script returned exit code 143
script returned exit code 143
kill finished with exit code 0
Sending interrupt signal to process
Killing processes
make: *** [dm_integration_test_in_group] Terminated
script returned exit code 143
kill finished with exit code 0
Sending interrupt signal to process
Killing processes
wait process dm-master.test exit...
make: *** [dm_integration_test_in_group] Terminated
script returned exit code 143
kill finished with exit code 0
[Pipeline] }
Cache not saved (inner-step execution failed)
[Pipeline] }
Cache not saved (inner-step execution failed)
[Pipeline] }
Cache not saved (inner-step execution failed)
[Pipeline] }
Cache not saved (inner-step execution failed)
[Pipeline] }
Cache not saved (inner-step execution failed)
[Pipeline] }
Cache not saved (inner-step execution failed)
[Pipeline] }
Cache not saved (inner-step execution failed)
[Pipeline] }
Cache not saved (inner-step execution failed)
[Pipeline] }
Cache not saved (inner-step execution failed)
[Pipeline] // cache
[Pipeline] // cache
[Pipeline] // cache
[Pipeline] // cache
[Pipeline] // cache
[Pipeline] // cache
[Pipeline] // cache
[Pipeline] // cache
[Pipeline] // cache
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // dir
[Pipeline] // dir
[Pipeline] // dir
[Pipeline] // dir
[Pipeline] // dir
[Pipeline] // dir
[Pipeline] // dir
[Pipeline] // dir
[Pipeline] // dir
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] // withCredentials
[Pipeline] // withCredentials
[Pipeline] // withCredentials
[Pipeline] // withCredentials
[Pipeline] // withCredentials
[Pipeline] // withCredentials
[Pipeline] // withCredentials
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // timeout
[Pipeline] // timeout
[Pipeline] // timeout
[Pipeline] // timeout
[Pipeline] // timeout
[Pipeline] // timeout
[Pipeline] // timeout
[Pipeline] // timeout
[Pipeline] // timeout
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // container
[Pipeline] // container
[Pipeline] // container
[Pipeline] // container
[Pipeline] // container
[Pipeline] // container
[Pipeline] // container
[Pipeline] // container
[Pipeline] // container
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // node
[Pipeline] // node
[Pipeline] // node
[Pipeline] // node
[Pipeline] // node
[Pipeline] // node
[Pipeline] // node
[Pipeline] // node
[Pipeline] // node
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] // podTemplate
[Pipeline] // podTemplate
[Pipeline] // podTemplate
[Pipeline] // podTemplate
[Pipeline] // podTemplate
[Pipeline] // podTemplate
[Pipeline] // podTemplate
[Pipeline] // podTemplate
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] // withEnv
make: *** [dm_integration_test_in_group] Terminated
script returned exit code 143
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] }
Cache not saved (inner-step execution failed)
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] }
Failed in branch Matrix - TEST_GROUP = 'G01'
[Pipeline] }
Failed in branch Matrix - TEST_GROUP = 'G02'
[Pipeline] }
Failed in branch Matrix - TEST_GROUP = 'G05'
[Pipeline] }
Failed in branch Matrix - TEST_GROUP = 'G06'
[Pipeline] }
Failed in branch Matrix - TEST_GROUP = 'G07'
[Pipeline] }
Failed in branch Matrix - TEST_GROUP = 'G08'
[Pipeline] }
Failed in branch Matrix - TEST_GROUP = 'G09'
[Pipeline] }
Failed in branch Matrix - TEST_GROUP = 'G10'
[Pipeline] }
Failed in branch Matrix - TEST_GROUP = 'G11'
[Pipeline] // cache
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Failed in branch Matrix - TEST_GROUP = 'G00'
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
Click here to forcibly terminate running steps
[Pipeline] End of Pipeline
org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 89f6060b-5236-4aa8-aa5d-c3ba9baea53a
Failed in branch Matrix - TEST_GROUP = 'G01'
Finished: ABORTED
![[Jenkins]](/jenkins/static/62b7ca6c/images/svgs/logo.svg){kind=logo}
