Skip to content
Success

Changes

Summary

  1. refactor(pipelines/pingcap/ticdc): optimize ticdc pipelines for resource (commit: 667fe50) (details)
Commit 667fe50e4910ab69eb300ec6972077f13cba2172 by noreply
refactor(pipelines/pingcap/ticdc): optimize ticdc pipelines for resource efficiency (#4109)

This PR optimizes the TiCDC integration test pipelines in
`pipelines/pingcap/ticdc/latest/` to improve resource efficiency and
consistency.

## The following changes were made

**Classic Pipelines:**

1. For `pull_cdc_kafka_integration_light` and
`pull_cdc_mysql_integration_light`, the pipelines were refactored to:
    - Use `agent none` at the pipeline level.
- Merge the `Checkout` and `prepare` stages into a single `Checkout &
Prepare` stage with its own agent.
    This ensures agents are only allocated when required.

2. For `pull_cdc_kafka_integration_heavy`,
`pull_cdc_mysql_integration_heavy`, `pull_cdc_pulsar_integration_light`,
`pull_cdc_storage_integration_heavy`, and
`pull_cdc_storage_integration_light`, the redundant stage-level
`timeout` option was removed from the `Checkout & Prepare` stage. This
centralizes timeout management at the global pipeline level.

**Next-gen Pipelines:**

All `next_gen` integration test pipelines were refactored to:
- Use `agent none` at the pipeline level.
- Merge the `Checkout` and `prepare` stages into a single `Checkout &
Prepare` stage with its own agent.

This aligns the `next_gen` pipelines with the resource-saving standards
of the classic pipelines, ensuring that agents are only allocated when
required. This reduces overall resource consumption and improves
maintainability.

## Tests

- [x]
[pull_cdc_kafka_integration_light](https://prow.tidb.net/jenkins/job/pingcap/job/ticdc/job/pull_cdc_kafka_integration_light/188/)
- [x]
[pull_cdc_mysql_integration_light](https://prow.tidb.net/jenkins/job/pingcap/job/ticdc/job/pull_cdc_mysql_integration_light/227/)
- [ ]
[pull_cdc_pulsar_integration_light](https://prow.tidb.net/jenkins/job/pingcap/job/ticdc/job/pull_cdc_pulsar_integration_light/177)
- [x]
[pull_cdc_storage_integration_light](https://prow.tidb.net/jenkins/job/pingcap/job/ticdc/job/pull_cdc_storage_integration_light/178)
(commit: 667fe50)
The file was modifiedpipelines/pingcap/ticdc/latest/pull_cdc_kafka_integration_light_next_gen/pipeline.groovy (diff)
The file was modifiedpipelines/pingcap/ticdc/latest/pull_cdc_kafka_integration_light/pipeline.groovy (diff)
The file was modifiedpipelines/pingcap/ticdc/latest/pull_cdc_pulsar_integration_light/pipeline.groovy (diff)
The file was modifiedlibraries/tipipeline/vars/cdc.groovy (diff)
The file was modifiedpipelines/pingcap/ticdc/latest/pull_cdc_kafka_integration_heavy/pipeline.groovy (diff)
The file was modifiedlibraries/tipipeline/vars/prow.groovy (diff)
The file was modifiedpipelines/pingcap/ticdc/latest/pull_cdc_pulsar_integration_light_next_gen/pipeline.groovy (diff)
The file was modifiedpipelines/pingcap/ticdc/latest/pull_cdc_storage_integration_heavy_next_gen/pipeline.groovy (diff)
The file was modifiedpipelines/pingcap/ticdc/latest/pull_cdc_mysql_integration_light_next_gen/pipeline.groovy (diff)
The file was modifiedpipelines/pingcap/ticdc/latest/pull_cdc_storage_integration_light/pipeline.groovy (diff)
The file was modifiedpipelines/pingcap/ticdc/latest/pull_cdc_storage_integration_light_next_gen/pipeline.groovy (diff)
The file was modifiedpipelines/pingcap/ticdc/latest/pull_cdc_mysql_integration_heavy/pipeline.groovy (diff)
The file was modifiedpipelines/pingcap/ticdc/latest/pull_cdc_mysql_integration_heavy_next_gen/pipeline.groovy (diff)
The file was modifiedpipelines/pingcap/ticdc/latest/pull_cdc_mysql_integration_light/pipeline.groovy (diff)
The file was modifiedpipelines/pingcap/ticdc/latest/pull_cdc_storage_integration_heavy/pipeline.groovy (diff)
The file was modifiedpipelines/pingcap/ticdc/latest/pull_cdc_kafka_integration_heavy_next_gen/pipeline.groovy (diff)