Continuous Testing Clarified, Part 3: CT and DevOps

Continue from Part 1 and 2.

Continuous Testing vs DevOps

“By 2020, DevOps initiatives will cause 50% of enterprises to implement continuous testing using frameworks and open-source tools.” — Predicts 2017: Gartner Report

If I ask you the hottest term in the software development industry in the past two years, many will say “DevOps”. Frankly, I think the term DevOps is quite vague (as opposed to ‘10-minute build’ and ‘pair programming’), therefore, it is open to interpretation.

Continuous Testing is the key (and most important) process of DevOps. You must have seen this DevOps image.

However, the below one is more accurate, as Continuous Testing is the pipeline with all the team members involved.

I have heard a few DevOps talks, however, they left no marks on my brain. For one project I witnessed in 2019, the executives got sold by the ‘impressive talk’ by a ‘DevOps talking-expert’, engaged the consulting company to implement DevOps. These consultants were busy talking, presenting, planning a roadmap, and introducing new software, …, for a few months. The result was a total disaster, in the end, the teams were told to revert back to the old way. The reason is simple: the foundation of DevOps is Continuous Testing. It is easy to understand, just imagine a pipeline producing poor-quality products in a factory. As we know, the quality problems in a pipeline magnify in order of magnitude.

Let’s switch the focus to DevOps’ objective (instead of its definition). I resolve to Wikipedia: “It (DevOps) aims at establishing a culture and environment where building, testing, and releasing software can happen rapidly, frequently, and more reliably”. This sounds quite like the objective of Continuous Integration, doesn’t it? Except with an emphasized focus on quality releases and feedback to the team. For example, if you set up a Jenkins or TeamCity to build software and run a few unit tests (i.e. programmer tests), you might call it CI, but it is incomplete in terms of DevOps, as it does not include regression testing (at the functional level) for releases.

I don’t mind DevOps at all. As a matter of fact, I have been developing software this way (releasing high-quality software frequently with comprehensive automated testing) since 2007, and have shared my experience in numerous presentations. Only at that time, the term “DevOps” and “CT” did not exist yet.

Continuous Delivery is really about testing

The origin of ‘Continuous Delivery’ is the book with the same title by Jez Humble and David Farley, published in 2009. Some will debate the differences among CI, CD and CT. Frankly, I don’t think it is necessary. The core of all three is the same: executing automated functional tests as regression testing.

Don’t just take my word for it, let’s hear the views from the authors of the Continuous Delivery book and a highly claimed authority in this field. In an interview in October 2019, Lisa Crispin, the co-author of the Agile Testing book, said this: “They (Jez Humble and David Farley) asked me to be a technical reviewer for their manuscript (Continuous Delivery book) … I read it. It’s a book about testing, you know, the whole book is really about testing. That’s the heart of continuous delivery. Jez Humble is very supportive of my saying that.”

Reality Check of CT/DevOps

Despite all the hype of CT (and previously CI/CD) and DevOps, the reality is 99%+ software teams at level 1 or 0 (a term I learned from the movie: Kungfu Panda) on the AgileWay Continuous Testing Grading. If you are not convinced, try to answer the two questions below:

• When was the last time your project pushed a release to production?
• How often do you do that?

In the context of DevOps, the correct answers for the above are “Yesterday or Today” and “At least once a day” (if changes were made).

According to this Forrester Study: Continuous Testing Separates DevOps Leaders From Laggards, IT executives often are not aware of immature/bad/fake Continuous testing practices in the company.

I have my reasons for using 99%+. Alan Page, the first author of “How We Test Software at Microsoft” book, said this at Test Talk PodCast #44, March 2015: “95% of the time, 95% of test engineers will write bad GUI automation just because it’s a very difficult thing to do correctly”.

Alan’s view has remained unchanged since 2008 when he wrote this on his blog:

“For 95% of all software applications, automating the GUI is a waste of time. For the record, I typed 99% above first, then chickened out. I may change my mind again.” — Alan Page’s Blog

Alan used ‘99%’ there, I added ‘+’ because most software companies won’t match Microsoft on

• quality of software test engineers
• resources (both technically and financially)

Furthermore, Continuous Testing adds more challenges, by running the whole test suite as automated regression testing frequently.