“Cucumber is Dying”, What did we learn? Part 2: FAQ

Continue from Part 1.

FAQ

1. Cucumber is just a Ruby implementation of Gherkin, maybe others such as SpecFlow in C# that work well?

People who hold this view need to catch up with Cucumber development in recent years. Since SmartBear acquired Cucumber Ltd in 2019,

Cucumber Ltd offers a full range of implementation, nearly in every major programming language, not just Ruby.

So, if the “dying Cucumber” is not due to a specific programming language, but rather, its core, which I explained in this article, “Why Gherkin (Cucumber, SpecFlow,…) Always Failed with UI Test Automation?”

2. How about other Gherkin frameworks, such as SpecFlow, not created by Cucumber Ltd?

To my knowledge, commercial-wise, SpecFlow and others are no different from Cucumber Ltd. The Gherkin syntax determines its fate. Even with a sponsor such as SmartBear, Cucumber did not work out, ….

3. Is BDD (Behavior Driven Development) about to die, too?

Firstly, Let me correct your assumption: BDD framework ≠ Gherkin. As we all know, Gherkin-style BDD started with Cucumber (in the Ruby language), but BDD Ruby started much earlier. The first and the most popular BDD framework in Ruby is RSpec.

BDD does not have to be “Given, When, Then …”. BDD with RSpec is great!

If your question is changed to “Is BDD with Gherkin about to die?”, my answer is Yes.

4. So, is this hugely-popular “BDD with Gherkin in Test Automation” a total fruitless hype?

Yes, it is. All end-to-end test automation attempts using Gherkin (Cucumber, SpecFlow, JBehave, Spinach, Gauge, Concordion), that I have witnessed, were all complete failures. 100%, with no exceptions!

The reason is so simple, Test Creation Only Account for ~10% of Web Test Automation Efforts, and the extra interpretation layer (for steps) makes test script maintenance much much harder.

5. Is there business value for BDD Gherkin in test Automation?

Based on SmartBear’s attempt, the answer is NO. Please consider the following:

1. SmartBear acquired the whole Cucumber, more importantly, the team including the founders.

This means, SmartBear, in theory, has the best knowledge of Cucumber, much better than some self-claimed BDD experts and book authors.

2. SmartBear invested a considerable amount, trying to earn money in the field, but failed.

From Matt’s “Every ending is a new beginning” post

Although our research showed there is a market for commercial tools built for the Cucumber audience, that market probably just isn’t big enough for a company like SmartBear. There’s no compelling reason for them to continue sponsoring an open source project that doesn’t directly contribute to the bottom line.

3. SmartBear is an established software testing company, yet it failed to use its resources to make Cucumber successful.

Regardless of my opinion of SmartBear, it has been a player (not a small one) in the test automation field for over a decade. If SmartBear failed to integrate Cucumber into/with its toolsets and succeed via its marketing resources (by the way, the study they did in 2019 suggested they could), then the chance for others to make Gherkin useful is very slim.

How could so many people get it wrong?

The answer is simple, only a very tiny percent of elite software engineers (including SET/SDET) can achieve End-to-End Test Automation success. Even LinkedIn needs to “lure” one (check out “The Software Revolution Behind LinkedIn’s Gushing Profits”). Don’t just take my word for it.

“In my experience, great developers do not always make great testers, but great testers (who also have strong design skills) can make great developers. It’s a mindset and a passion. … They are gold”. - Google VP Patrick Copeland, in an interview (2010)

“95% of the time, 95% of test engineers will write bad GUI automation just because it’s a very difficult thing to do correctly”. - this interview from Microsoft Test Guru Alan Page (2015)

“Testing is harder than developing. If you want to have good testing you need to put your best people in testing.”- Gerald Weinberg, in a podcast (2018)

My daughter started her first intern job (freshly after uni) at a large telecom company in 2021. She learned test automation (raw Selenium) + Continuous Testing casually during high-school breaks. I told her that even though she is only Level 2 of AgileWay Continuous Testing Grading (not that good, in my eyes), she would probably still be the best test automation engineer this large tech company has ever seen. My daughter, only 20 years old, didn’t believe it at all. It turned out to be exactly as I predicted. Check out the story: An IT Graduate’s frustration with a Fake ‘Senior Test Automation Engineer’.

Later, she asked me why? I answered: “End-to-End Test Automation is so easy to get wrong. You did OK because you learned it the right way (raw Selenium + RSpec), without much effort”. Using Gherkin in end-to-end test automation is an example of wrongdoing. Also, as I have witnessed over a decade, once a bad decision in test automation, such as QTP, Phantom.js, Protractor, Cypress, …, etc, was made, few acknowledged it after its failure. Most commonly, people pretended nothing happened, back to manual testing, until the next hype. Check out the article, “Correct two Common Misconceptions: End-to-End Test Automation is ‘Simple and Easy’ or ‘Complex and Impossible’”.

What lessons do we learn?

The Test Automation industry always has big false hypes all the time, to name a few:

• Record-n-Playback tools, such as HP QTP
• Cross-Browser Testing
• Headless Test Automation with Phantom.js
• Before Gherkin, there is Fitnesse (its webpage is no longer available, the link is to Wikipedia)
• Protractor
• Cypress
• …

Many ‘automated testers’ got lost in all kinds of hypes, confused and unable to move forward. By the way, my predictions on test automation hypes have been always accurate (with proofs):

• Record/Playback in Test Automation is Bad, mostly
• Cross Browser Testing Clarified
• All those who claimed to be “Automated Testers” and Promoted “Headless Testing” were Fake
• A Practical Advice on Rejecting Gherkin for Test Automation
• Protractor, another Automation Framework I rightly avoided, is being Deprecated
• Why Cypress Sucks for Real Test Automation? (Part 2: Limitations)

While I have been doing successful end-to-end test automation since 2005. Prior to Selenium WebDriver, I used Watir+RSpec, then raw Selenium WebDriver + RSpec. No fancy talks, just get real tests implemented on the first day, add more the next day, while always maintaining all valid every day onwards.

--

This article was originally published on my Medium blog on 2023-05-05.