Continuous Testing Clarified, Part 2: CT vs CI

Continuous Testing is the most important process in DevOps; Yet most so-called ‘Agile’ or ‘DevOps’ projects did not have it or got it wrong.

By Zhimin ZhanPublished 10 months ago • Updated 10 months ago • 3 min read

Continue from Part 1.

We cannot talk about Continuous Testing without comparing it to “Continuous Integration” (CI in short). IT staff often get confused between these two terms: CI and CT, some even use them interchangeably. Here, for simplicity, I will point out one key difference:

CI: executing unit and integration tests
CT: executing automated End-to-End (preferably, via UI) tests.

Continuous Testing vs Continuous Integration

Continuous Integration is “where members of a team integrate their work frequently, usually each person integrates at least daily — leading to multiple integrations per day”. In this famous CI article’s original version (published in 2000. By the way, that’s how CI started), Martin Flower used “often talked about but seem to be rarely done” in the first sentence. Based on my observation over the last two decades, this still remains true: the term CI is favoured by “talkers”.

CI Reality

I remember at one CITCON (Continuous Integration and Testing Conference) in 2009, a delegate talked about why he attended the conference: “I want to know how other projects are doing CI? The closest CI experience I ever encountered was that one machine was assigned to do it, then ticked the box. No one touched the machine again.” Many agreed with him.

A decade later, most software claimed “doing CI” is no more than building (code) and deployment (package), with little or no execution of automated tests …

Here I highly recommend a great presentation: “Continuous Integration at Facebook”.

If CI is implemented properly, no need for CD or CT

CI has been so messed up in practice that it is becoming meaningless. That is why a new term comes up “Continuous Delivery” (CD in short, which later quickly lost its meaning as well), somehow people find it fancier to say these two terms together “CI/CD”. In every project I visited over the last decade, agile coaches/architects talked a lot about CI/CD and did not do hands-on test automation, their continuous integration processes were all embarrassing failures.

Once I worked at a software company, they had a Bamboo CI server with a number of projects, which seldom ran and virtually no sign of executing automated tests. However, they claimed they were providing CI consultancy to one of the top four banks in Australia.

If CI’s main purpose is to build a releasable software package, this has been achieved years ago (before CI concept existed) with build scripts, like an Ant task generating a deployable war file (back to J2EE days). Triggering a build from a web interface and seeing build results (on the CI server) is nice, but don’t you think there is not much to brag about? The purpose of CI is to ensure quality releases by running automated tests against release candidates. The testing, in particular, End2End user story level testing, is the main part.

Some people might say “Continuous Testing” could be the next ruined ‘talker term’. Yes, that could well be true, and probably already is. At this moment, we have run out of terms, sadly. I will settle with the term “CT” for now, because of its emphasis on testing.

So what is the relationship between CT and CI? In simple words, CT is a part of CI, the most important and difficult part. If a CI process is implemented properly, there is no need for “CD” or “CT”.

In Part 3, CT and DevOps.

Further reading