Benefits of E2E Test Automation & Continuous Testing (Part 2: to Wise Managers)

E2E Test Automation & Continuous Testing (enables software teams to push software updates to production daily, not fake CI/CD talks. Check out “Continuous Integration at Facebook” and AgileWay Continuous Testing Grading) is the heart of the software development process. It benefits all stakeholders from a software project.

• Executives
• Managers
• Business Analysts
• Developers
• Testers (automated & manual)
• Customers

Software managers deal with day-to-day software activities which can be quite stressful, as we know. When a project commences, a manager usually refers to the Project Management handbook for planning, risk assessment…, etc. A few weeks in the project, all that theoretic project management stuff would have been thrown away, as the project manager would be either in feel-like-endless meetings or consumed by putting out fires such as defects.

Unrealistic Expectations

A typical software project manager, as I have observed, seemed to live in a perfect world initially, when everything goes as planned in Gantt or Velocity charts. However, the reality is that software development is full of uncertainties, such as:

• Scope creep (also known as ‘requirement creep’ or ‘feature creep’)
• Change of requests
• Defects
• Staff change

It will be painful for project managers to deal with differences between the dream (Gantt or Velocity Charts) and reality (defects and tech debts) daily.

Automated Testing is far beyond just verification

I have met a few project managers who believed that the project team should compromise quality (reflected by functional testing, i.e, quality assurance) to meet deadlines. This is very wrong.

Once I asked the audience during a presentation: “Which country or culture do you associate with the highest quality?”. “Japan” and “Germany” were the two answers from over 100 audiences. My next question was, “Which country or culture do you associate with the highest efficiency? ”, the answers were “Japan” and “Germany” again.

This simple experiment tells us that high quality can lead to high efficiency/productivity, and it is especially true for software development. For example, the widely-accepted code refactoring practice is a process of continuously refining the quality of the design (backed by automated unit testing) to improve on-going development efficiency.

In other words, a wise IT manager weighs more efforts in the efficiency of automated testing to gain the team’s high productivity.

Continuous Testing is a manager’s “best friend”

A good Continuous Testing process will greatly improve the project manager’s quality of work-life (and night-sleep). The reasons are simple:

• CT prevents and minimizes a lot of uncertainties, such as defects. A software issue detected earlier means a lot easier and cheaper to fix, and prevent other problems. A project with a successful CT process might not need defect tracking. What a relief for managers!
• Knowledge sharing among the team with automated tests. How did this feature work? Find its automated test and run it; When a new staff joins, let him run a suite of key tests to get familiar with the system quickly (at his own pace); What is the login for the manager user? Get it from automated tests; …
• Team management is much easier. With a good CT process, the team members have a common goal: green build. To a degree, “One team, one dream” becomes a reality.
• CT greatly increases the team’s productivity. A project manager (who loved Gantt charts) did measure the team's productivity after I introduced the test automation to the team. He told me that overall productivity had increased by 300%. For my own web app development (since 2012), the productivity gain is at least 10x after a CT process is in place. See my article “Reflections of Software I Created over the Last 14 Years in My Spare Time”.

Wise project managers will still do the planning but will be much less serious about Gantt/Velocity charts because they see real progress every day, and they know the team is travelling much faster and better (than the planned drawn in the pre-CT days) anyway. “On time, on budget” is a sure thing now. The question becomes “How much can we do better than customers’ expectations?” Gradually, the wise manager’s main focus is moving to how to improve the CT process further.

I personally have seen a couple of PM have these transformation. They were amazed how impactful the CT is. One PM wrote this in a letter to a senior executive (he showed me the letter): “I am experiencing a revolution. I know you will find what I said next is hard to believe, but it is true ….”

I once worked with a senior project manager who had a programming background. He was impressed with my development process: writing automated functional tests for each user story and keeping all regression tests pass by the end of the day. He pointed at the regression testing report on BuildWise, “This makes me sleep well at night.” He had two young sons, and he said this to me: “After I have been working in the IT industry so long, I didn’t want them to do IT when they grow up. Now I see how you work, I changed my mind”. This comment, in my opinion, is the best compliment of my Continuous Testing work.

Challenge to introduce CT

1. Find qualified test automation and CT coach

Assign your best engineers a task to develop/maintain about 50 automated tests, then measure their performance against AgileWay Continuous Testing Grading. If it is Level 0 or 1 (most likely), your team does not have the skills. Seek help from a qualified test automation and CT coach.

Your next question will be: “How do I know whether a coach is fake or not?” Easy, ask him to work on real tests, and then measure his work against AgileWay Continuous Testing Grading. In just one or two days, you will know. Most of the time, you can even tell in just a few minutes. I have seen several ‘senior test automation engineers/coaches’ hands shaking on the keyboard and they gave up. They were only good at talking about test automation and CT.

Under the mentoring of a real test automation and CT coach, your engineers shall be able to work on real tests efficiently right the next day. From my memory, there is no cost of introducing test automation, as the manual tester under mentoring will be more productive the next day. Generally speaking, a regression suite of 50 end-2-end tests is achievable. Please note, these tests (existing and new) shall be executed and pass every day. Then the coach only needs to come in as needed, 3 days per week, 1 day per week, or per month. Then you will have a team with CT capability. If there is an unexpected challenge, you will have peace of mind knowing the trustworthy help is not far.

2. Prevent sabotage

Apart from technical challenges, there are challenges from human factors. Mediocre people fear changes, and CT will certainly make many fuzzy things clear and transparent.

In most cases, the test automation and CT (or CI/CD) in the company have failed a few times already. Even so, some technologies (and tools) are still the company's so-called ‘strategy’. For example, have you ever seen one successful Continuous Testing of a reasonable sized automated UI regression suite in Jenkins? I haven’t, not a single case. But many companies named Jenkins as their CI/CD tool. The same goes for the test automation framework, the test syntax framework, and the test scripting language.

To suggest alternatives (if you have read my book or other articles, my suggestions are: use raw Selenium WebDriver in RSpec, and run them in the free BuildWise CT server), a manager needs to be courageous and tactful.

Advice on introducing CT to your project

Propose a 3-day proof of concept (POC) implementing Continuous Testing, which will usually be accepted by the upper management. Do the POC well and real. Choose the best technologies (such as raw Selenium WebDriver + RSpec) that will work, ignore the company’s so-called ‘long-term Agile/DevOps strategy’ (if it was correct, all teams would have been able to do production releases daily).

If you were approached by the DevOps (or Agile Transformation) team, just send them the report (test count, builds per day, test pass rate) of POC, and politely ask their help to implement their recommended approach, to achieve the same results. Of course, the DevOps team won’t be able to do it. Therefore, they will probably leave your project alone. If you had the DevOps team involved early before the POC, most likely, you would be dragged into many meaningless meetings, without any outcomes.

To do a real successful (that most of your team members see the value, not just a few simple tests) Continuous Testing in 3-day POC, you may need support from a technical expert. So, prepare well. Either work with your best engineer or seek external help from a Continuous Testing coach. Being practical is the key. As a manager, you shall monitor the build report frequently and even run a few automated tests now and then.

Advice on project management with CT

Here I share the approach by the best Agile project manager I have met; his management style is incredibly simple and effective:

1. Only he uses JIRA, other team members only work with the printed-out user story cards. No team members (except him) is aware of (or care about) the team’s velocity.
2. Every developer needs to show and run at least one Automated End-2-End test to him. Only by then, this user story can be marked done.
3. Schedule two CT builds (12 PM and 5 PM) every day, but not limited to. Team members may trigger builds as necessary.
4. A failed build becomes the top priority task.
5. If a red build (test failures) in the CT server lasts more than 24 hours, there will be no meetings and no work on new user stories.

--

The original article was published on my Medium blog, 2021-03-12