Changes

ticdc: Add next-gen integration tests for Kafka, MySQL, Pulsar, and Storage. (#3831)

This pull request introduces new Jenkins pipeline files for running
integration tests for TiCDC. These pipelines are designed to run on the
"next-gen" infrastructure, indicated by the `_next_gen.groovy` naming
convention.

The primary motivation for these changes is to expand the integration
testing capabilities for TiCDC by adding support for Pulsar and
improving the existing Kafka and MySQL integration tests.

Here's a summary of the changes:

*   **New Jenkins Job DSL Files:**
    *   Added `pull_cdc_kafka_integration_heavy_next_gen.groovy`
    *   Added `pull_cdc_kafka_integration_light_next_gen.groovy`
    *   Added `pull_cdc_mysql_integration_heavy_next_gen.groovy`
    *   Added `pull_cdc_mysql_integration_light_next_gen.groovy`
    *   Added `pull_cdc_pulsar_integration_light_next_gen.groovy`
    *   Added `pull_cdc_storage_integration_heavy_next_gen.groovy`
    *   Added `pull_cdc_storage_integration_light_next_gen.groovy`

*   **New Jenkins Pipeline Files:**
* Added
`pipelines/pingcap/ticdc/latest/pull_cdc_kafka_integration_heavy_next_gen/pipeline.groovy`
* Added
`pipelines/pingcap/ticdc/latest/pull_cdc_kafka_integration_heavy_next_gen/pod.yaml`
* Added
`pipelines/pingcap/ticdc/latest/pull_cdc_kafka_integration_light_next_gen/pipeline.groovy`
* Added
`pipelines/pingcap/ticdc/latest/pull_cdc_kafka_integration_light_next_gen/pod.yaml`
* Added
`pipelines/pingcap/ticdc/latest/pull_cdc_mysql_integration_heavy_next_gen/pipeline.groovy`
* Added
`pipelines/pingcap/ticdc/latest/pull_cdc_mysql_integration_heavy_next_gen/pod.yaml`
* Added
`pipelines/pingcap/ticdc/latest/pull_cdc_mysql_integration_light_next_gen/pipeline.groovy`
* Added
`pipelines/pingcap/ticdc/latest/pull_cdc_mysql_integration_light_next_gen/pod.yaml`
* Added
`pipelines/pingcap/ticdc/latest/pull_cdc_pulsar_integration_light_next_gen/pipeline.groovy`
* Added
`pipelines/pingcap/ticdc/latest/pull_cdc_pulsar_integration_light_next_gen/pod.yaml`
* Added
`pipelines/pingcap/ticdc/latest/pull_cdc_storage_integration_heavy_next_gen/pipeline.groovy`
* Added
`pipelines/pingcap/ticdc/latest/pull_cdc_storage_integration_heavy_next_gen/pod.yaml`
* Added
`pipelines/pingcap/ticdc/latest/pull_cdc_storage_integration_light_next_gen/pipeline.groovy`
* Added
`pipelines/pingcap/ticdc/latest/pull_cdc_storage_integration_light_next_gen/pod.yaml`

*   **New Prow Job Configuration:**
* Added `prow-jobs/pingcap/ticdc/latest-presubmits-next-gen.yaml` to
integrate the new pipelines into Prow.

* **"Next-Gen" Infrastructure Support:** All new pipelines are
configured to run on the "next-gen" Jenkins infrastructure, leveraging
updated pod templates and artifact download scripts.

* **Integration Test Coverage:** The new files provide dedicated
pipelines for testing TiCDC's integration with Kafka, MySQL, Pulsar, and
various storage backends in both "light" and "heavy" test
configurations.

* **Standardized Pipeline Structure:** The new pipelines follow a
consistent structure, including stages for debugging, checkout,
preparation (downloading dependencies and building binaries), and
running tests.

* **Matrix-Based Testing:** The 'Tests' stage utilizes a matrix strategy
to run tests in parallel across different `TEST_GROUP` values, improving
test execution efficiency.

* **Artifact Archiving:** Failed test runs will archive relevant logs
for easier debugging.

---------

Signed-off-by: tenfyzhong <tenfy@tenfy.cn>
 (commit: feb81c9)