Two Simple Reasons that ChatGPT (AI) Won’t Work in End-to-End Test Automation. Part 2

Contiue from Part 1.

2. Test Creation is only a minor effort in Test Automation

What is the most challenging in end-to-end test automation? The answer is "Test Maintenance". Check out this story, a survey (I personally involved) from ~70 testers in a company.

Based on my estimation, test creation only accounts for 10% of the whole test automation effort. The other two are "Stabilization/Refinement" and "Ongoing maintaining". This is logical, as software changes constantly and frequently in agile projects, i.e. lots of maintenance work. While people may debate 10% or 20%, test creation is definitely minor. Otherwise, the old record-n-playback tools, such as QTP, will still dominate test automation.

ChatGPT, if it indeed helps (which I don't think so, see this article), is on test creation. In a big scheme, it does not matter much, anyway.

Suppose somehow an AI Testing tool helps you create a good suite (200+) of automated tests. After your team tweaked and verified all the scripts, the application changed. How will your team keep those tests valid? Before you answer, remember these facts:

• Your team did not know the test scripts (partly created by AI tools) well.
• One single application change (even an extra comma in a JS/CSS file) might cause an unknown number of tests to fail.
• The changes are constant and frequent. This means some of these end-to-end tests (created by AI) will be outdated.

Unless AI is smart enough to magically get all test scripts updated correctly on your command "Fix all broken tests", I don't see the usefulness of AI in end-to-end test automation.

Think about it. If "AI can fix all broken tests" in seconds, there will be no need for testers, i.e. you are out of jobs (and won't be able to find another tester role, forever).

Summary

I think software engineers and test automation engineers, who are super-hyped about ChapGPT or other AI, are silly. Codeless programming (powered by so-called AI), in particular, was not new and failed.

Some will argue that the new AI (such as ChatGPT) will change everything. Maybe or maybe not. I listed my reasons why ChatGPT won't be helpful in end-to-end test automation.

This hype, in my opinion, is like previously failed ones in end-to-end test automation:

• Record/Playback will make developing automated end-to-end tests easy and effortless. Very wrong!
• Object Identification GUI Utility will make test maintenance easy. Totally wrong!
• Headless testing with PhandomJS. Wrong, PhantomJS was deprecated in 2017.
• BDD using Gherkin (e.g. Cucumber and Specflow) in test automation.  Oh well, 12 years passed…. Ironically, the creator of Cucumber warned, "what is Cucumber? As a test tool it sucks."
• Protractor.js on the ride of super popular Angular.js. Protractor was deprecated in 2021.
• After Protractor's failure, a new wave of JS-based test automation frameworks, such as TestCafe, Puppeteer and Cypress. Cypress.io (the company behind Cypress) is dying. To my knowledge, many JS testers have been dumping Cypress and moving to Playwright (which is not good either, compared to raw Selenium WebDriver with Ruby).

The only proven working solution is raw Selenium WebDriver, which existed 12 years ago. To get it working, test automation engineers need knowledge, right-tool-sets, good test-design skills and the mindset of ongoing maintenance.

"Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released." - DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.

Below is a recent run of my WhenWise app's regression suite, consisting of 571 raw Selenium WebDriver tests.

There is no sliver-bullet. Hypes of 'snake oil' remedies have consistently been proven false.

If you want to be a real test automation engineer, my advice has been the same since 2011: "Learn and Practice".

• Learn Selenium WebDriver with a scripting language such as Ruby for web testing.
• Practice daily. Check out these exercises.

That's the quickest way, if done properly, it will be quite an engaging, fun and rewarding journey.

---

The original article was published on my Medium Blog, 2023-02-23