Estimating Test Automation Story Points is a Total Waste of Time

Many fake “agile project managers/agile coaches” are fixated on Velocity Charts. The so-called “velocity” is based on the rate of completing the story points estimated by the team. As a result, “story estimation sessions” are a part of “Agile Ceremonies” in some projects, along with“stand-up meetings”, “retrospectives”, and “sprint planning”. (see the end of the article for Kent Beck’s, the father of Agile, view on these ‘agile ceremonies’)

I do not estimate points/time for developing automated tests for a user story. The reason is simple: it does not make sense.

Below is a slide from a presentation I gave in 2015.

It has three charts:

1. A typical Scrum Sprint: 10 working days (2 weeks)

The planning sessions are on the first day, and the showcase & retrospective are on the last day. Software development activities (coding, testing) occurred on the 8 days in the middle.

2. A typical testing effort for Sprint #2

Assume the team is trying to do Agile, and a completed user story is verified by one or more automated end-to-end tests (“Done, Done”). From Sprint 2 onwards, the testing activities consist of the following:

• Prepare the automated tests for this sprint
• Develop the new automated tests
• Regression, make sure the automated tests created in the previous sprints are still valid (common sense, right?). Besides test execution, this includes test maintenance, such as updating test scripts/data along with application changes, stabilizing test execution, and refining/refactoring test scripts.

3. The regression effort will accumulate quickly along with the development.

On Sprint 3 (or even earlier), the regression testing effort will exceed all other test automation activities.

When I showed the above charts one by one, most of the audience showed signs of agreement with me. However, the natural conclusion is quite different from their previous thoughts, that is, the maintenance efforts of previous tests will far exceed that of developing new automated tests of the current sprint. Then I said, “So, it is meaningless to estimate the efforts on automated-test creation for new stories, as the majority of test automation engineer efforts are on regression and maintaining existing tests”. From my memory, no single disagreement from the audience.

Two simple facts about UI test automation:

1. UI test execution takes longer time (much longer than unit/api tests)

2. one simple application change may affect an unknown number of automated tests.

I could sense that most of the audience was shocked by the finding, and the reasoning was so strong that it was hard to debate. One of them asked: “I concur, but it will be hard to convince managers”.

Check out this article, A Practical Advice on Rejecting Estimating Story Points for Test Automation.

Some might wonder, "why so many did not figure it out before?"

The answer is simple: most agile coaches/scrum masters/tech leads have never worked on a real Agile project. They knew test automation was an essential part of Agile, but had never experienced a real test automation process that overcame the hard-to-maintain phase. Commonly, test automation was for a showoff at the start of a project, then given up with all sorts of excuses. Executing all automated end-to-end tests as regression testing multiple times a day is a new concept to them. One of my colleagues told me: "Running automated regression testing daily is how test automation shall be done. Why haven't I thought about this on other projects?

This is by no means belittling my colleagues (we got along quite well). This is the failure of IT education and training. I was fortunate enough to receive proper training from a world-class hands-on agile technical coach in 2005, after working as Senior Java Programmer (Contractor) for 8 years. That was the turning point of my IT career.

--

The original article was published on my Medium blog on 2021-04-21