Why Don’t I Use Defect Tracking? No Need, I do real Continuous Testing.

Replicate a new defect with automated tests, then add it to the regression suite that runs in a Continuous Testing process.

By Zhimin ZhanPublished 9 months ago • Updated 9 months ago • 5 min read

Those teams who are spending hours every day on a defect-tracking system (DTS), such as Quality Center, may get offended by the title of this article. Here, I borrow quotes from some of the world's leading experts to support my view.

The answers to track/fix the bugs, from these three highly-claimed books, are the same: creating an automated test to replicate the bug, so that you don’t need to log it in DTS. Here, I would like to share my own experience of practising this in the past decades.

“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.

The bugs/defects I am referring to are the ones found by the IT team members. Those that are reported by the customers, of course, will be treated with high priority and may be recorded. For my own apps, I usually implement customers' feature/change requests or bugs within the same day, so I don't track customers' reported defects either. This is a fact, otherwise, I wouldn't have been able to develop and maintain 6 complex apps:

TestWise, next-gen functional IDE
BuildWise, an international award-winning Continuous Testing Server
ClinicWise, a cloud-based practice management system
SiteWise CMS, content management system
SupportWise, our internal support system
WhenWise, an online service booking system

all in my spare time.

I worked on a government project in 2008. During a stand-up meeting, the project officer mentioned this, "I received a renewal invoice for XXX (a defect tracking software) ...". The team members paused for a few seconds as most of them had never heard of this DTS. It turned out that the project purchased an annual licence for a defect tracking software before the development started (government does that), but we had never used it. A young developer declared: "but we don't need to do defect-tracking". Period.

Defect tracking is a costly process. A senior project manager once told me that 15% of his project budget was spent on Defect Tracking and Triage (prioritizing defects based on severity, frequency, and risk) processes. Quite often, I heard that some projects bear hundreds of (at one extreme case, over 1000) defects. If this is the case, defect tracking will not be helpful in fixing defects. The process of defect tracking will not only consume time and budget but also create frustrations and anxiety.

Let me be clear: though I do not like defect tracking, it certainly is not the reason to discard it. I avoid defect tracking by eliminating the need for it. Believe it or not, after having worked on a real agile project (with comprehensive automated end-to-end regression testing), I found raising defects is unnatural in software development. (I could not remember the last time I raised a defect, while I worked as a software testing engineer in a number of projects. On those projects, my automated test scripts detected more defects than other testers combined, via regression testing in CT. The defects were shown on the CT Server, and I did not log them in DTS). Certainly, during the development of my own apps (since 2007), I have never used defect tracking.

Let's examine why defect tracking is unhelpful and unnecessary:

1. Defect Tracking is an illusion

When a tester raises a defect, it does not guarantee that the defect will be fixed, nor a sure repeatable way to prevent that from happening again. I know some will argue about this, please read on. It simply created an illusion that this issue will be dealt with, which, again, is not guaranteed.

2. The steps to replicate the defect are quite often incorrect

The objective of doing defect tracking is to record the steps that lead to a possible issue. However, it only works in theory. We, as humans, make a lot of mistakes in documenting a list of steps in an exact fashion.

I once worked on a software testing project for a long-living system (over 20+ years). Its test design document (with written test steps and test data) was understandably large and detailed after years of revisions. However, a large percentage of test cases were simply not executable for newbies. Experienced testers were able to perform testing mostly based on their past knowledge rather than the written test case design.

3. Logging defects in DTS is 'lazy', delay the fixing process

Every software engineer knows the best time to fix a defect is now. The cost and effort will be exponentially higher the next day, as the engineer's mind is already on another task. Quite often, programmers, who prefer fixing defects in one go at the end of spring, would usually introduce new defects on bug-fixing. I have seen it many times, the Continuous Testing process picked up those newly-introduced defects quickly. Just imagine a team without CT, how bad is that! No wonder some complain 'software development is stressful!'

4. Promotes 'blaming culture'

How often have you heard of a programmer saying, "It worked on my machine" or "It is a feature, not a bug"? I have worked as both a programmer and a tester. Therefore, I have seen countless debates on defects between programmers and testers, which brought conflict to the team.

5. Pointless debating a defect's severity

Have you seen a debate between two team members on which severity this newly-found defect shall be? I have, for quite a number of times. These debates were often pointless. With over 100+ severity 2 defects in DTS, team members usually won't be able to fix them for a long time. Why bother?

6. Inefficient tracking regression errors

One regression issue may be detected many times during development. Do you raise the same defect multiple times or just one with a long list of notes? As you see, this will complicate things.

7. When there are so many defects, tracking is meaningless

The tracking process will eat the team alive.

8. "Broken Window" theory

If there are hundreds of outstanding defects, a programmer tends to be careless when developing new features or fixing bugs. It is the famous "Broken Windows" theory, featured in the classic "Pragmatic Programmer" book.

9. Low team morale

Obviously, it is hard for an engineer to take pride in a product with hundreds of defects.

How to avoid these defect-tracking shenanigans? Simple: Implement a real Continuous Testing process so that you don't need to track defects at all. When a defect is detected, replicate it in an automated test case. This new test joins existing tests as the regression testing suite, which will be run in a Continuous Testing process multiple times a day.

Of course, if the Continuous Testing process is not set up (or done poorly) for your project, Defect-Tracking is still the sensible way to do it, but at a high cost.

Here are some stats of CT for one of my web apps: "WhenWise Regression Test Suite Reaches 500 Selenium tests and ~300K Test Executions". If you are interested in my approach to Continuous Testing, check out my book "Practical Continuous Testing".