Continuous Testing Clarified, Part 1: What is CT?

Continuous Testing is the most important process in DevOps; Yet most so-called ‘Agile’ or ‘DevOps’ projects did not have it or got it wrong.

By Zhimin ZhanPublished 9 months ago • Updated 9 months ago • 3 min read

This article is one of the “IT Terminology Clarified” series.

Continuous Testing, according to Wikipedia, is “the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.” The keywords are “automated tests”, “delivery pipeline”, “immediate feedback” and “software release candidate”. (The reason I quote Wikipedia, a non-academic reference, for definitions: because Continuous Testing is new and there is a lot of confusion over it. So I reach for a common understanding at Wikipedia)

Continuous Testing explained in simple words

Like many other formal technology definitions, the above (on Wikipedia) sounds right, but is not quite clear. Let me interpret it:

“executing automated tests”, “business risks”. The automated tests are at the user story level: testing business features. For example, for a web app, the CT process will run a set of automated test scripts to drive the app to verify business functions, in a browser. Comparatively, unit testing is conducted by programmers only, that’s why unit tests are also called programmer tests.
“pipeline”, “immediate feedback”. In this pipeline, Customers/Business Analysts and programmers are waiting for feedback, and more importantly, ready to act on the feedback. Modification follows feedback, i.e. if there are test failures, a new build (with potential fixes) will be triggered to run another execution of automated tests to ensure the quality.
“software release candidate”. The software is in a ready-to-release state if it has passed all automated functional tests, with little or no human activities in pushing the latest release to production.

Here is my interpretation of Continuous Testing: “Run automated end-to-end (UI) as regression testing, frequently on new builds. If all tests pass, the software is ready for a production release. If there are test failures, the team must act quickly on the feedback.”

For the benefits of End to End Test Automation and Continuous Testing, please check out this article series.

Continuous Testing is the Holy Grail of Software Development

I didn’t invent using ‘Holy Grail’ to describe CT (as much as I would like to take credit for, I heard it from a conversation years ago, but pitifully, could not remember the source), but I think it is a perfect metaphor for Continuous Testing. Yes, it is a big claim, and many of you probably are doubtful. It is my hope to convince you with the Practical Continuous Testing book.

Here I share one customer’s comments after I helped implement the continuous testing process, which enabled the project to push updates to the production server on a daily basis. This product owner agrees: “Continuous Testing is super valuable and super rare, despite many heard of it, but very few saw it, just like the Holy Grail”.

Prior to practising test automation and CT, I have worked as a senior software engineer (Java and C#) for a number of years. I did not create a single my own product in spare time. Now have a handful highly-acclaimed software products, one even won an international programming award. Check out this article: Reflections of Software I Created over the Last 14 Years in My Spare Time,
When someone asked my secret, My answer: “I found the holy grail (of software development), that is E2E Test Automation and Continuous Testing”.

Software Development Trend ⇒ Continuous Testing

The above is a summary of key advancements in the software industry over the last 25 years. As you can see:

Testing Method: Manual → Automated
Testing Type: Unit => End-to-End
Testing Frequency: Sporadic (every a few months) → Frequent (daily, or even a few times a day)

Please read this article for detail.

This article was originally published on my Medium blog, on 2021-11-16.

In Part 2, Comparing CT vs CI.

In Part 3, CT and DevOps

Further reading