Protractor.js is dead, What did we learn? Part 2

Continue from Part 1.

4. A test automation framework/tool from a high-profile company does not mean it will be successful.

The Angular team (at Google) had the utmost reputation during its peak time. People have reasons (kind of real) to believe that its test automation framework will be successful, just like its core product, Angular.

In reality, big software companies had a very poor record on test automation products:

• HP QTP (being acquired by Micro Focus, now rarely heard)
• IBM Rational Function Tester (RFT) - I haven’t heard of it since my review at a government job, in 2013.
• Microsoft Coded UI Test, deprecated in 2018.

I often see “create a test automation framework” listed in Job Ads for test automation engineer roles. Isn’t that crazy (even Protractor.js team at Google failed)? Check out my article, Please, Not Another Web Test Automation Framework, Just Use Raw Selenium WebDriver.

Real test automation engineers are extremely rare, even at Google and Microsoft.

“In my experience, great developers do not always make great testers, but great testers (who also have strong design skills) can make great developers. It’s a mindset and a passion. … They are gold”.

- Google VP Patrick Copeland, in an interview (2010)

“95% of the time, 95% of test engineers will write bad GUI automation just because it’s a very difficult thing to do correctly”.

- this interview from Microsoft Test Guru Alan Page (2015)

“Testing is harder than developing. If you want to have good testing you need to put your best people in testing.”

- Gerald Weinberg, in a podcast (2018)

Also, check out this story, “The Software Revolution Behind LinkedIn’s Gushing Profits”.

Yes, please notice, it is “Lure”, not “Hire”. LinkedIn should know how to hire people, right? 😊

Some might disagree, surely, there are at least a few real test automation engineers at Google. Yes, as Google Senior VP said, “they are gold”. What I said the above are facts, because I didn’t work at Google. The following is my assumption. Real test automation engineers would instantly know raw Selenium WebDriver is the best, wouldn’t bother joining this Angular/Protractor team. By the way, Simon Stewart, the creator of Selenium WebDriver, worked at Google during that period.

I often say a software engineer will be extremely lucky to meet on real test automation engineer in his/her career. I was very fortunate to meet one in 2005 and learned test automation and Continuous Testing from him faithfully and hungrily. Check out My Test Automation Journey.

How did I predict its failure?

I have acquired knowledge from years of years daily working on real and useful end-to-end test automation, which enables teams ‘daily production releases’.

It is quite easy to create a fancy demo of a so-called new ‘test automation framework’, which I am sure you have seen before (I saw too many). But that’s far far from a real test automation framework, like Selenium WebDriver. For most test automation attempts, the teams were unable to maintain 50 test cases (Level 2 of AgileWay Continuous Testing Grading) daily running correctly. Then, what’s the use of test automation? Test Automation is the foundation of Agile/DevOps, without that, Agile or DevOps was simply fake.

By the way, if a team can keep 20 tests (Level 1 of AgileWay Continuous Testing Grading) daily running correctly, that is still limited useful. However, sadly, most teams get 50+% failures (kind of intermittently in different test scripts) daily. For example, I started noting down the test failure rates on starting client projects since 2012, the best pass rate (with ~50 tests) on their CI servers is 48%. The target rate, according to Definition of End-to-End Test Automation Success: 100%, which I helped acheived a handful with raw Selenium WebDriver + RSpec.

For example, Cypress (another so-called JS test automation tool) started promotion even without supporting ‘frames’ (and other limitations), which many enterprise websites use. This is totally insane, especially the best and feature-complete Selenium WebDriver was already there. It is like selling a car that it will melt on a rainy day!

If the test automation attempt at your company just failed, and you want my advice on a test automation framework/tool, raw Selenium WebDriver + RSpec, in the wonderful Ruby language (‘the most in-demand skill’, according to Hired’s 2023 State of Software Engineers Report). For tools and CT, check out AgileWay Test Automation Formula, it is proven and you don’t have to pay a cent.

--

This article was originally published on my Medium blog on 2023-05-12.

Further reading:

• Testing Pyramid Clarified
• Test Automation Camel, a metaphor that explains why most test automation attempts failed?
• My new “Web Test Automation with Selenium WebDriver Training Workbook” series on Substack. The most practical and fun way to learn real test automation, suitable for 10+, with no prior knowledge (coding/testing/automation) at all.
• 35-Word Functional Test Automation Strategy