Regression Testing Clarified, Part 2: Why few projects do Automated Regression Testing

Continue from Part 1.

Most IT people agree with the benefits of regression testing, implemented with E2E Test Automation. However, in reality, few software projects do it.

I recommend the executives/managers have a realistic assessment of the capability and openness of your engineers. Surely, daily automated regression testing (desired by most CIOs) will have a huge impact on software development. Wired magazine wrote this in the article “The Software Revolution Behind LinkedIn’s Gushing Profits”: “completely overhauled how LinkedIn develops and ships new updates to its website and apps”. It is worth repeating: “completely overhauled”. If an IT executive thinks: “Buying an expensive tool or engaging some expensive consultants will get CT implemented”, they are dreaming.

From my experience, after initial hesitation or objection, most engineers would thank you for making this happen. Once implemented, life would be better for everyone.

“This (Continuous Integration, in the context, the authors means ) is a difficult discipline, partly because developing a rapid build and test capability requires time, skill, and ongoing attention, and partly because stopping to fix problems can be distracting. But a great epiphany occurs once a software development team finds ways to do rapid builds and continuous integration and everyone faithfully stops to fix problems as soon as they are detected. We have been told many times that this is a very difficult discipline to put in place, but once it is working, no one would ever think of going back to the old way of doing things.”

- Implementing Lean Software Development: From Concept to Cash, by Mary Poppendieck and Tom Poppendieck

6. Why don’t software teams do automated regression testing?

The short answer is that the tech leads usually don’t understand test automation, or worse, they think they do. I was in that position (~2005) before; it took me about 2-years of learning and practising test automation (at work during the day and at home during the night). Since then, I have been doing hands-on test automation and Continuous Testing every working day (for my clients and my own apps).

Let me share a story.

Many years ago, I joined a large .NET project as a contract software engineer. Technically, the project was a mess. To give you a feel, the configuration engineer (whose main role is to build the software package) once yelled with excitement: “It compiled!”.

The time I joined was after the first internal release, and many quality (functional) issues were reported. The star programmer Rob, who was a senior technical consultant at a well-known international software consulting firm, proposed automation, especially for regression. A few weeks later, an internal demonstration meeting was arranged.

The newly joined project manager, who worked with me on the previous project, suggested that I show my approach. I accepted, which I later thought was a mistake. For this reason, I remember this session quite well.

In the meeting, Rob demonstrated his approach to regression testing:

• Invoke a test to send input to the application (a workflow product), and save the output to a file, e.g. case1–1209.txt
• On the next build candidate, repeat the above to save to a different output file.
• Run a diff tool to compare these two files; if they are the same, no regression issues.

I was quite shocked, this naive ‘regression testing’ demo was well received by other team members. The tech lead even said, “cool”.

Then I came up and showed my regression testing, which I used for my own work for a while. I picked up a case like Rob’s. Different from Rob’s regression testing approach, mine is simpler:

• Verify every step inside the test scripts

If all steps pass, the regression run of the test case passes! And I showed a report of the execution history (daily for ~3 weeks) of my test suite (~50 test cases).

There was a moment of silence after my demo. Clearly, everyone realized this was real regression testing, as it did real verification. Comparing the output approach was no good:

• if there is timestamp info, the output will never be the same
• adding an extra print statement will cause ‘regression failure’
• we have no idea (from examining the output) which step failed.
• there were no assertions, i.e., not testing.

They probably were embarrassed (for their reactions to Rob’s demo) after seeing my demo. The PM was wise to notice the awkwardness. He stood up and quickly wrapped up the meeting.

Of course, nothing changed after the meeting. I still did my own regression testing; No one talked about automated regression testing anymore. However, it seemed there was a gap between other engineers and me. That’s why I said it was a mistake to show my approach.

It is not those engineers' fault, though. It is the failure of IT education and training. We have programmers studying for 3 or 4 years for an IT bachelor's degree but barely touch on functional test automation and continuous testing. But in a typical software team nowadays, the number of testers often doubles the programmers' count. And strangely enough, very few software companies reach for external help on training and coaching test automation and CT.

Further reading:

• 35-Word Functional Test Automation Strategy
• Testing Pyramid Clarified
• Benefits of Test Automation and Continuous Testing series: Executives, Managers, Business Analysts, Developers, Testers and Customers.