Regression Testing Clarified, Part 1

This article is one of the “IT Terminology Clarified” series.

Between 2010–2013, I spoke at a few international software testing conferences (and decided no more, see reasons here). Quite commonly, there were pre-attendance surveys on the attendants’ interested topics. From my memory, the top three (the order might vary) were the same.

• Test Automation (End-to-End)
• Regression Testing
• Web App Testing

However, topics on regression testing were rare at software testing conferences. Several attendants told me this at different events (after attending mine).

In this article, I will share my thoughts and experience in regression testing. This article is a bit long, with the following sections:

• What is Regression Testing?
• Status of Regression Testing in Software Projects (quite sad)
• Regression Testing must be automated, however, …
• Manual Regression Testing is Evil in Agile
• How often do you do regression testing? (correct answer: multiple times a day)
• How long should full regression testing take? (correct answer: about 30 minutes)
• Why don’t software teams do automated regression testing?

For readers who are interested in the implementation, check out this article: How to Implement Real Automated Regression Testing?

1. What is Regression Testing?

Regression Testing in software means “re-running functional and non-functional tests to ensure that previously developed and tested software still performs after a change” (Wikipedia). Every software team member understands the objective of regression testing, as we often see regression errors. However, not many software engineers (including testers) understand how to do regression testing properly.

2. Status of Regression Testing in Software Projects

Most software projects conduct regression testing manually. As a result,

• Long feedback loop - Commonly, manual regression testing takes a few weeks or months in some software projects.
• Expensive - Many human resources and a long time, which means high costs in software development. Also, there are costs associated with the server/resources tied to a long regression testing cycle.

• Limited Coverage - Due to the high cost and time constraints (of manual regression), regression testing coverage is small, often getting smaller. In the end, regression testing becomes limited to the business features the team thought it had been impacted by the new changes. Of course, it is wrong. The nature of software development is that a change might impact unknown parts of software, that’s why we need regression testing in the first place.
• Management overhead - Besides major releases, there are minor updates and production fixes. All these require regression testing, some do it on a much small scale. The risk with production fixes may be small, but risks are risks. I prefer no risks if possible, by a proper and fast regression testing process. (I always do full regression testing because I do it efficiently with automation.)
• Low Morale for testers - Obviously, nobody likes to do repetitive tasks, after one or two times.

With so many shortcomings, why do most (my estimation is >95%) software projects still do manual regression testing?

3. Regression Testing must be automated, however, …

"The only practical way to manage regression testing is to automate it" - Steve McConnel in the class book "Code Complete" 2nd edition, page 528

There are many reasons such as lacking test automation skills and experience in managing test executions in a Continuous Testing (note, not CI) server. Often, “IGNORANCE IS STRENGTH”, engineers were faking test automation and CT. As a result, we see this often in so-called ‘agile’ projects:

• a handful of test automation engineers work on something no one knows or cares about.
• A large group of manual testers conduct manual functional testing slowly. Most of their efforts are spent on regression testing.

Of course, there are no excuses for such costly and inefficient regression testing in agile projects. But most managers/executives chose to be ignorant.

4. Manual Regression Testing is Evil in Agile

Manual Regression Testing is unfavorable, not just due to its substantial costs and taking a very long time, but also because it leads to a decline in the quality of testing.

"Test guru Boriz Beizer reports that the error rate in manual testing is comparable to the bug rate in the code being tested. He estimates that in manual testing, only about half of all the tests are executed properly." - Steve McConnel in the class book "Code Complete" 2nd edition, page 528

The best adjective I can relate to manual regression testing is “CRUEL”, the best noun would be “SWEATSHOP”. The best adjective for manger and tech leads, who let manual regression testing happen, is “INCOMPETENT”. One purpose of using software is automation, how can we let poor manual testers do these repetitive tasks again and again?!

If a project’s regression testing is mostly done manually, it surely is NOT Agile.

“Using JIRA does not mean Agile; End-Users do not care story points or velocity charts. They don’t mind not-perfect new features, but they will get annoyed if existing features were broken by updates. So, as software professionals, we need to do regression testing properly” — Zhimin Zhan

5. How often do you do regression testing?

Many software professionals often provide a vague answer to this question. Generally speaking, it depends on how often you push releases to production.

(The below are two typical production release cycles in real agile projects. I exclude fake agile projects in which a production update occurs every 1+ months. My advice to those projects is to drop everything and review their process seriously. Stop estimating story points is a good start)

1. Every Iteration (sprint in Scrum), typically 2 weeks

2. Every day, typically twice a day

“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — from the presentation of Damien Sereni (engineering director at Facebook) at Selenium 2013 Conference

BTW, I have practised daily releases for over 12 years. When I got a green build in my BuildWise CT server, I pushed the build to production immediately.

6. How long does it take?

You might have heard of “10-minute build” in some Agile talks. The origin of ‘10-minute build’ is the classic (and the first) Agile book by Kent Beck, the Father of Agile.

But Kent did not say specifically what a build means. There are several understandings:

1. Compile and Package, such as .war file for a Java app
2. Compile, Unit Testing and Package
3. Compile, Unit Testing, Code Coverage, Integration/API testing and Package
4. Compile, Unit Testing, Integration Testing, Package, Deploy and E2E (UI) testing

My take is № 4. If a green build (pass all tests) in a CT server, I can push the build to production. For most software companies, a 10-minute build (of №4) seems impossible, but it can be achieved.

How could Facebook achieve (not always though, I strongly recommend watching the presentation) that? The picture of its parallel testing lab below might answer the question.

Facebook takes automated regression testing seriously, as it enabled “release twice a day”, which is “the heart of its culture” (see the quote earlier). By the way, Facebook uses the free Selenium WebDriver. So, don’t be fooled by the salesmen of expensive commercial tool vendors such as Micro Focus, Tricentis, SmartBear, …, etc. The best is free.

Some engineers might be overwhelmed by Facebook’s efforts and determination (from the top executive) in Test Automation and Continuous Testing. Tech leads, before you come up with excuses, there is possible that small and medium-sized IT companies can achieve good regression testing time (around 30–59 minutes) with a very tight budget (~$200 to $400 per month).

In Part 2, I will explain why few projects do automated regression testing.

Further reading:

• My eBook: “Practical Web Test Automation with Selenium WebDriver”
• My eBook: “Practical Continuous Testing: make Agile/DevOps real”
• Testing Pyramid Clarified