Benefits of E2E Test Automation & Continuous Testing (Part 5: to Testers)

E2E Test Automation & Continuous Testing (enables software teams to push software updates to production daily, not fake CI/CD talks. Check out “Continuous Integration at Facebook” and AgileWay Continuous Testing Grading) is the heart of the software development process. It benefits all stakeholders from a software project.

• Executives
• Managers
• Business Analysts
• Developers
• Testers (automated & manual)
• Customers

I have full respect for manual testers as I have gained a lot of insight into good automated test design by working closely with these testers as a consultant, programmer or mentor. The bad manual testing processes that some organisations force on poor manual testers are just cruel.

The Reality

A typical test case design/execution spreadsheet looks like below.

Every time when I see this, the word ‘sweatshop” comes to me. How could we say that we are doing High-Tech if our testers spend most of their time doing these tedious and mostly unuseful tasks day after day? There are many problems with this type of test execution:

• Written Test Steps are often out of date. From my memory, > 90% of manual test designs (Excel style) will be wrong after a while. The decision to create these test spreadsheets (quite a lot of effort) is based on the assumption: the application won’t change. However, we know that this assumption is completely wrong, especially for ‘Agile’ projects. Some might argue: “If the application changes, we will update the affected test steps in Excel”. Will you really do this every time? I doubt it as I have never met anyone who does diligently.
• Test Data quickly out of date. Compared to test steps, test data in Excel became obsolete much more quickly. For example, for the test case “17-year-old client requires to fill extra ..”, the first test step is “Login as c1024576”. However, this client might not be valid next time: the client is now over 17, the account is no longer invalid or does not exist in another environment. Some will say: “I would write this: find a client account that is under 17 years old”. However, a new manual tester won’t be able to execute the test based on the test steps then. This is just a simple example.
• Manual testing based on this is often executed wrongly and poorly

• Regression testing is practically impossible if done manually

• Slow regression tests mean: Real Agile/DevOps is simply impossible. How long will it take to run the full regression testing (based on a comprehensive test spreadsheet)? From my knowledge, a typical answer is about 1 to 3 months for a serious web app. There might be answers such as “5 days for key regression”, but that is not counted as comprehensive regression testing. If a round of full regression testing takes more than a day to run, the project should say ‘goodbye’ to DevOps. Why? We usually find defects in regression testing (if it always passes 100%, why bother to do it?), which means another round of regression testing for bug fixes, …

The Hope

Software testing can be done in a much more efficient, creative and fun way: E2E Test Automation (via UI). In a recent project, I introduced test automation to the team and the manual testers (4 of them) were doing test automation with high spirits. In the farewell email from the lead developer who observed the changes, he wrote: “I appreciate you giving our testers a new lease on life.”

However, there are two common major concerns a manual tester has for test automation initially:

• “It is hard”

Not really. Test automation can be quite easy to get started. Both test automation and continuous testing can be learned in gradual steps. The hard part of maintaining a good number of automated tests.

Under proper mentoring, testers may apply newly learned automation skills to work quickly. Most of the testers I mentored could develop a few automated test scripts for real work test cases on the first day. Check out this article: Step by Step showing how to learn to write raw Selenium WebDriver test scripts in minutes.

You probably wouldn’t believe that you could set up a CT server from scratch to run selenium tests within 30 minutes. But it is possible. Check out this article: Set Up a Continuous Testing Server to Run Selenium Tests in Minutes.

• “It may replace my job”

This is not true. Agile advocates more testing (naturally with more frequent releases) and automation. Compared to 20 years ago, I find there are usually more testers than programmers in an ‘agile’ team. Besides, the development of an automated test script starts with manual testing. Manual testing, especially in exploratory testing form, will be always needed. With continuous testing in place, testers might perform exploratory testing while a build is in progress.

“Having said that, it is important to note that Google performs a great deal of manual testing, both scripted and exploratory, but even this testing is done under the watchful eye of automation”. —“How Google Tests Software”, page 7

After repetitive and tedious regression testing has been taken care of by CT largely, testers will now have time and tools (utilising automation scripts) to facilitate high-quality manual testing.

The Help

When test automation is introduced, some manual testers want to learn and some don’t. That’s OK. As I have said before, exploratory testing and scripted manual testing are still required, though less.

Anyhow, all testers (plus other team members such as business analysts) can utilize automation scripts to increase testing efficiency and generate test data. For example, a manual tester is about to test a user story “Claim manager can reject an insurance claim application”. Instead of manually filling in a new claim application, she could run an automation script to do that part. After the execution (of the automation script) is completed, she could pick up right from there in the same browser window.

--

The orginal article was published on my Medium Blog on 2021-08-26