Contributed by Ole Lensmar, Testkube, and Tracy Ragan, DeployHub
New Testing Tools – Old Testing Workflows
Over the years, testing has evolved well beyond a team of Quality Assurance experts running software and hunting for bugs. Testing now spans the DevOps pipeline from coding through runtime, with various testing goals supported by a broad selection of testing tools. As the type of testing activities increased, so did the need to automate the testing pipeline. Hundreds of testing pipelines have been developed to execute complex testing tasks, often called sub-workflows to the Continuous Integration/Continuous Delivery (CI/CD) pipeline. In other cases, stand-alone testing pipelines have been developed to meet the needs of different environments.
The result is thousands of statically defined testing workflows that must be maintained and updated to stay relevant as new testing technology becomes available. In addition, as organizations move from monolithic to microservice architecture, testing processes must address decoupled components where hundreds of services are being built and deployed independently and consumed across multiple clusters and “logical” applications. Adding new testing capabilities to old testing workflows is a barrier to adopting testing innovation. This is the problem that the CDEvents project addresses.
According to the CD Foundation’s CDEvents Whitepaper (PDF) (May 2023):
“The CDEvents project’s mission is to standardize an event protocol specification that caters to technology-agnostic machine-to-machine communication in CI/CD systems. The CDEvents project aims to provide reference implementations such as event consumers/ listeners and event producers/senders on top of for example CloudEvents.”
Introducing Testing Events to CDEvents
The CDEvents team has recently added Testing Events to the CDEvents specifications under the Testing Events bucket, to simplify the automation of testing processes from code through runtime.
Testing Events have the potential to revolutionize the adoption of testing practices by making it simple to add different testing tools across hundreds of pipelines. Testing events also address the need to standardize the collection of test results as well as critical metadata such as where the logs are stored, regardless of the test tool/framework, which enables visualization and centralizes the data produced during test execution in a more “federated” way.
Work Completed
The CDEvents team has established specifications for three subject areas:
| Subject | Description | Predicates |
testCaseRun | A testCaseRun is a process that executes a test against an input software artifact of some kind, for instance, source code, a binary, a container image or else. A testCaseRun is the smallest unit of testing that the user wants to track, testSuiteRuns are for grouping purposes. | queued, started, finished |
testSuiteRun | A testSuiteRun represents the execution of a set of one or more testCaseRuns. | queued, started, finished |
testOutput | One or more testOutput artifacts are usually produced as the result of a test execution. | published |
To manage Test events common objects have been defined, including:
- TestCase – A testCase is the actual test that is being run by a testCaseRun.
- TestSuite – a collection of
testCaseobjects managed in an external system. Each time atestSuiteis executed the correspondingtestSuiteXXXevents should be emitted. - Trigger -What started a corresponding testCaseRun/testSuiteRun.
For a full description of all of the above, visit the CDEvents Testing GitHub Repo.
Moving from static to dynamic Testing Workflows
By introducing the above-mentioned subjects and common objects in the CDEvents specification, CD workflows can now be dynamically extended to include new testing activities as applications evolve and new testing practices or tools are added to the CD process. Instead of manually having to update statically defined testing workflows, testing and orchestration tools can listen for CDEvents to dynamically trigger the execution of newly created tests and output the corresponding Test Events defined above to allow downstream systems to react accordingly, for example by promoting an artifact to a new deployment environment when a set of newly introduced tests passes.
What now?
Building a more agile DevOps Pipeline requires improved ways to integrate tooling into the pipeline workflows and the CDEvents project team continues to be laser-focused on building a common data sharing specification that can be used as a basis for these integrations. As we have learned from our community, the inclusion and automation of testing activities is critical to these pipelines, and we are happy to see that CDEvents now specifies Testing events that express the occurrence and context of testing activities and their routing from producer to consumer.
Help us improve testing
The CDEvents team encourages participation in the CDEvents conversation and looks forward to working with member companies, open source projects, and interested professionals to help build an event-based CD Pipeline that can easily adapt to the changing DevOps landscape.
Join the Team
Visit CDEvents.dev to learn how you can join the CDEvents team and help build the next-generation CD Pipeline.
Learn More
Watch our CD Pipeline Techstrong.TV episode on testing: Advancing Testing in Continuous Delivery: Unveiling Challenges, Strategies, and the CD Foundation’s Initiatives
