Benefits of E2E Test Automation & Continuous Testing (Part 6: to Happy Customers)

E2E Test Automation & Continuous Testing (enables software teams to push software updates to production daily, not fake CI/CD talks. Check out “Continuous Integration at Facebook” and AgileWay Continuous Testing Grading) is the heart of the software development process. It benefits all stakeholders from a software project.

• Executives
• Managers
• Business Analysts
• Developers
• Testers (automated & manual)
• Customers

“On-site Customer” is one of the agile practices. However, this practice seems to have been largely forgotten in recent years. Besides human factors, I think the main reasons are:

• Low development productivity. It is not practical to ask an on-site customer to wait 9 working days (a typical sprint length is 2 weeks) to see some new features implemented. Customers shall be able to try out the application and provide feedback quickly and frequently.

By the way, the 2-week sprint, in my opinion, is too long. I recommend 2 sprints per week; if CT is implemented, it is really not that hard to achieve it.

• Poor software quality. When a software team lacks a good automated regression testing process, they will encounter unexpected defects often. Under these circumstances, project managers will naturally want to hide the product from the customers and ignore the ‘on-site customer practice.’

The customers only care about the product. Imagine you are a customer representative who is assigned to work with the software team. You take time out of your normal work to attend many meetings that talk about story points, velocity and burn charts in JIRA. Understandably, you, as well as most on-site customers, would be pissed off with those non-sense so-called “agile ceremonies”.

Agile is all about feedback, the agile practices only make sense if the customers get to use the software product to try out new features quickly. Of course, the major challenge is to to introduce regression errors. That’s why test automation and Continuous Testing are the foundation of Agile. While most agile coaches/scrum masters agree, few of them have actually seen one real Test Automation and CT, let alone implementing one.

Take showcases as an example. A showcase is an opportunity for the software team to show what business functions they have accomplished and demonstrated them to the customer for feedback. However, I have noticed showcases have gradually changed over the years:

• Fewer teams actually do showcases,
• Little or no live demonstrations, which is the main purpose of Showcases. (fake) Scrum masters spent a lot of time on useless stuff such as burn charts.

Though showcases have become dull and boring, every single time when I ran automated tests against the real server, I could feel the atmosphere in the meeting room come alive. The real stuff is what people want to see, including the team members.

While the benefits of Showing business features via automation regularly may not be easy to quantify, keeping customers impressed will definitely boost the customer’s confidence in the product.

On one project, we used test automation and CT regularly in the showcases. A regression testing report (exported from the CT server) was made available for the customer at the end of each sprint. During the warranty period, the CIO of the customer company told the project manager: “Send me a BuildWise report of a green build, then you have my permission to deploy the update to the production”.

When the customers really trust the software team’s professionalism, everything is easier, a lot easier.

My Experience

This happened in a real agile project with daily-production-release capability enabled by Continuous Testing.

Story: Customer Demonstrated newly implemented features

There was a particular showcase that was done by a customer representative. You read it right. It was this customer that demonstrated newly implemented features in the showcase. Thanks to our CT process, the team rolled out features quickly and addressed most customer feedback within 2 days. In this way, the customer was naturally involved more. She saw the test executions in the CT server (BuildWise) and even ran specific tests in testing IDE (TestWise).

After this on-site customer demonstrated all features implemented (last spring) in one showcase, she continued and said: “There is a new important feature we are also working on, coming in the next sprint”. She demonstrated a few steps, and an error occurred (the error stack trace shown in red on the screen). But she was still smiling: “As I have said, this is a work-in-progress, the guys will fix this shortly.” Tim, the developer who worked on this new feature, noted this down solemnly (he probably could not wait to add an automated test to replicate the error).

What impressed me the most was when that error occurred, the atmosphere in the room was still relaxed. Because we all knew, including the customer, the error would soon be captured in a new automated test, and then fixed.

Showing Test Automation to Customers makes sense

How do we measure quality in everyday life? Testing. The fundamental reason that we feel safe to take aeroplanes is that we trusted the engineering behind. There is a strict process of manufacturing, servicing, testing and maintenance of planes (including staff training). An investigation (often formed by multi-parties) will be conducted when there is an unfortunate civil plane accident. If there is a fault in the mechanic or the operating process, the identified issues (made public) will be addressed to prevent the same from happening again.

Let me show you another example.

This is from a luggage advertisement, which says:

• “wheels 360 ° tested to 40km.”
• “tested to durability limits, including extreme drop tests, tumble tests …”

In software development, customers rarely see a proper engineering process. They see programmers write code with headphones on, managers talk and show useless charts, and testers manually verify the software unenergetically according to a long spreadsheet document. The team spends more time on Jira than on its own product. In a word, customers don’t see the quality or a trustworthy process.

Some IT managers or engineers might think: “This is just how it is in IT”. No, this only appears normal in mediocre software companies.

Let’s have a look at another example: Facebook’s software testing lab.

This is why customer involvement is totally changed (for the better, of course) when Test Automation and Continuous Testing is implemented. You just have to experience it to really appreciate it.

My advice to software customers: “Adding test automation and Continuous Testing into the contract, the deliverables are code and E2E tests; All E2E tests are required to run on a daily basis”.

If the software team can do real test automation and CT, you will be happy. If the software team cannot, which means the development would have failed early. In that case, you will probably be happy too, for the trouble you avoided.

--

The original article was published on my Medium blog on 2021-08-30