A Long List of Failed Test Automation Hypes

this is a concise version of the original article, published on Medium (2021-04).

Since I have worked in software development and test automation for over two decades, I have seen many failed hypes in test automation, such as:

• Record-n-Playback Test Automation Tools, such as HP QTP (now Micro Focus UFT) and IBM Rational Functional Tester.
• Cross-Browser Testing using SauceLabs or BrowserStack, Nowadays, people care little about CBT (focus on Chrome only).
• Headless Testing with PhantomJS, an embarrassment for many, I correctly predicted its failure (written in my book).
• Protractor (JS), the testing framework based on Selenium from the super hot Angular.js team, deprecated in 2021.
• Test Cafe, Puppeteer and Cypress. Those three have been or are being replaced by Playwright, which is no good either, compared to the best Selenium WebDriver. Cypress.io is dying.
• Codeless Test Automation. Do you see anyone using it for real?

The above are the failed automation hypes. If we extend to include test syntax:

• FitNesse. Many people don’t hear it now, Back in ~2008, it was a quite big hype.
• Gherkin Styled BDD (Behaviour Driven Development). Cucumber is Dying.

I predicted the failures of all the above (and others), with proven records: my books (since 2009), my tools (since 2007), and my articles. Since 2011, I have been implementing successful end-to-end (via UI) test automation exactly the same way: raw Selenium WebDriver + RSpec ✅. Prior to Selenium WebDriver (released in 2011), I used Watir + RSpec (for executing tests in IE).

Check out, Showcase a 500+ End-to-End (via UI) Test Suite: E2E Test Automation

1. Record-n-Playback Test Automation Tools

Reason for failure: Hard to maintain Test Scripts.

The recorder-generated proprietary test scripts are terrible: difficult to read, and almost impossible to maintain when the test suite reaches a certain size.

Check out the article, Record/Playback in Test Automation is Bad, mostly.

2. Cross-Browser Testing Tools

Reason for failure: No signs of daily execution on Chrome, little need for worrying about others.

Also, there was clearly a trend that Chrome (or Chromium) would extend its dominance in the browser market. IE was dying (update: dead), and Edge Classic sucked (update: dead) and is now based on Chromium.

By the way, Selenium WebDriver is best for Cross-Browser Testing, the reason is simple: All major browser vendors only support WebDriver, a W3C standard.

Check out the article, "Cross Browser Testing Clarified".

3. Headless Testing with PhantomJS

Reason for failure: Not a real browser, how could handle ever-growing complex web apps. Furthermore, more often than not, we want to see test execution.

Between 2010–2015, Headless browser testing with Phantom JS was hyped highly by test architects/engineers. I was puzzled about how come people could not see the obvious wrongness.

If a test script fails on a simulated browser (not setting the web page), how do you debug it?

PhantomJS was deprecated in 2017, and suddenly, headless browser testing disappeared (until a few years later, Google Chrome implemented headless mode in v58).

Check the article, "Headless Browser Testing Clarified".

4. Protractor (JS)

Reason for failure: Making intuitive Selenium syntax complex, injecting Programmer’s thoughts into test automation.

Despite of super-hype of Protractor.js (by Angular.js team), it failed (deprecated in 2021). It made good raw Selenium syntax bad based on wrong knowledge of ‘improving’ it. When I saw a sample Protractor test for the first time, I felt like seeing a clean room that was badly decorated.

Check out the articles:

• Protractor, another Automation Framework I rightly avoided, is being Deprecated
• Protractor.js is dead, What did we learn?

5. Test Cafe, Puppeteer, and Cypress

Reason for failure: a web test automation framework must support all web technologies defined by W3C. But these don’t.

I know Cypress’s UI (as a tool) might appear appealing and good for demonstration. That’s all. Check out my other articles:

• Why Cypress Sucks for Real Test Automation?

• Comments on the claims of “7 ways Cypress is different”. All False, Wrong or Lie
• Exit Strategy for Cypress Testers

Making a wrong framework decision in Web Test Automation is NOT excusable.

Web Test Automation means Objective and Quick Results. Why? Testing means objective; Core Web technologies have not changed over two decades.

“If a test automation engineer has ever done one successful test automation, the knowledge, skills and practices can be directly applied to future ones. It is like “Bike Riding”, once you really master it, you prove it instantly. “ — Zhimin Zhan

It shall be quite obvious after being pointed out (see the above), isn’t it?

---

Related reading:

• Why Raw Selenium Syntax is Better than Cypress and Playwright? (3 parts)
• Why Ruby is the Best Scripting Language for End-to-End Test Automation?