Implementing Continuous Testing in DevOps Pipelines

7 min readMay 11, 2021

Co-written with Sudhish Mangalasary

Organizations continue to modernize their software development and delivery practices to minimize the impact of business disruption and stay competitive. Organizations have implemented Agile techniques to accelerate software delivery and are increasingly turning towards DevOps to improve their success and productivity by implementing a continuous delivery model.

Speed of Software Delivery and Quality

Many of them have adopted continuous integration (CI) and continuous delivery (CD) pipelines. However, they have soon realized that there is no longer a choice between the speed of delivery and quality. Organizations need both as they are key differentiating factors. Continuous testing as an integral part of the pipelines can bring back focus on quality in a fast-paced delivery model. This great blog by Marc Hornbeek on “Continuous Testing: The Quest for Quality at Speed” sums it up well.

These five tenets from Hornbeek:

Shift Left: Conduct each test as early in the pipeline as possible
Fail Early: Arrange the tests so the most likely problems are found during the earliest possible stage in the DevOps pipeline
Fail Often: Run tests frequently and with many different conditions
Test Fast: Arrange tests to run in quick cycles
Be Relevant: Focus on the most important tests and results

form the basis for any team looking to really leverage continuous test to drive quality at speed. Now how does this apply to DevOps pipelines and the automation around that?

Continuous testing ensures to deliver quality into the DevOps pipeline by constantly running manual and automation tests. Continuous testing saves cost, time and resources if implemented appropriately in pipelines, with the added benefit of advancing DevOps pipeline maturity. It provides continuous feedback to stakeholders in every stage of pipelines to take necessary actions or decisions. This will be realizable only if the culture in an organization follows Marc’s five tenets and places a priority on that, a priority that does not wax and wane over time.

However, there are quite a few challenges in implementing Continuous Testing in Pipelines:

Best of breed tools: no single solution provides support for a complete toolchain to build CI /CD pipeline
Scaling up and managing the complexity of tests: tests grow both in number and complexity with the maturing code and the environment becoming more complex
Creating feedback loops: providing visibility and information in every stage of a pipeline
Availability of environments: the need for multiple environments for various test scenarios.
Dependency on other teams: waiting on the schedules and efficiencies of other teams such as development and deployment

Best practices that can help in implementing and improving quality throughout the development and release cycles are below:

Adopt Lean testing: Collaborate with business and other stakeholders to build meaningful and important end-to-end test scenarios.
Have combined ownership of pipelines: both developers and testers should monitor the pipelines, troubleshoot and fix bugs in cognizance of deployment lead time.
Avoid duplication of tests: same tests running in multiple test suites delays releases.
Manage tests and automate only what is needed: Automating every test can have a higher maintenance cost as any change in feature can result in test regressions.
Don’t leave failing tests unattended: And at the same time, evaluate the usefulness of tests that never fail.
Use manual tests wisely: have manual tests in the pipeline especially when exploratory tests and usability tests need to be performed when features are under development.

The ReleaseIQ platform simplifies the creation of an effective continuous test pipeline in three easy steps and is designed to unify teams across the value chain by providing end-to-end visibility and actionable information for all stakeholders in every stage of the pipeline. Our platform provides flexibility to optimize pipelines adhering to best practices while preserving the history of previous runs and pipeline definitions.

Continuous Test Support in the ReleaseIQ Platform

The steps for creating pipelines for continuous testing in ReleaseIQ are:

1. Continuous Test Support in the ReleaseIQ Platform

Select tools and provide details for secured ReleaseIQ agent access.
Define multiple Environments for deployment and testing in any stage of the Pipeline.
Specify Test results formats.
Junit, TestNG & Chromium are processed and rendered with no additional coding or configuration.
For Custom test results in XLSX, XML and CSV formats, map the columns in the test results file to ReleaseIQ attributes as shown.

Alternatively, plugins can be developed to convert Test results to ReleaseIQ native format as per ReleaseIQ XML schema and deployed to process the Test results.

2. Compose Pipelines

Create CI or CI/CD pipelines as needed to embed quality assurance processes in DevOps pipelines for continuous testing and feedback.
Tests can be invoked by
Jobs in CI tools defined in Settings. Select from the dropdown list of available jobs.
Shell scripts programmed in the Custom Script as shown.

Set email notifications on test success, failure or both.
Specify the percentage of test failure tolerance as gating criteria for the pipeline to proceed on failure or fail.
Create continuous integration (CI) pipelines for each component level or for each microservice in the modern application as needed.
Integrate tests(Ideally automated test) for more frequent integration of builds.
Run tests for every code change to find and fix bugs earlier in the development cycle.
Create continuous deployment (CD) pipelines that are an extension of CI pipelines that helps in releasing integrated updates from individual CI pipelines at the production-end of development.
Integrate tests (manual and automated tests) to ensure code change is releasable.
Run tests to reduce the risk of releasing new builds to Users as quickly as possible.

3. Monitor & Troubleshoot Pipelines

Monitor pipeline status and lead time with personas-based perspective views.
View summary of the pipeline runs or get a detailed view of the pipeline runs for every code change to ensure relevant tests are run.

Test summary of each Test suite can be viewed by clicking the Testsuite bar. Test start time, duration, test case details, and associated logs are also available.

In case of test failures, Troubleshoot with logs available in ReleaseIQ UI that help to resolve issues quickly to avoid delay in releases.

File bugs in Jira for issues for tracking them to closure. This can be done directly from ReleaseIQ UI or Jira UI.
The test team can provide progressive updates on manual tests. The tests results in XML, XLSX or CSV format that previously configured in Settings can be uploaded.
Allows multiple file uploads. Especially when test duration spans over days, the updates can be provided on a daily or on a frequent basis for the Development team to comprehend corrective action quickly.
The history of all uploads of test results is maintained. The test team can mark manual test complete by ticking the “Test Cycle Complete” checkbox.

Once the Manual Test is marked complete the pipeline progresses based on the fault tolerance specified.
Test stability view provides details of test runs over a duration with details of test case results for each run that help identify tests that frequently change statuses or consistently fails. Two different views are available to drill down to details of each run.
Summary view: Donut chart view of Tests in the order of the greatest number of tests runs over a specified duration

Heat Map View

The end result of creating these pipelines is a seamless transition from CI/CD to more quality-focused CI/CD delivery model, with no additional effort in providing end-to-end visibility of pipelines and feedback to all stakeholders in an organization.

Want to learn more about our integrations and how to configure, run and gain unified visibility for all your pipelines, watch our recent webinar or view our product brochure.