A Practical Advice on Rejecting Gherkin for Test Automation

Gherkin BDD frameworks (such as Cucumber, SpecFlow, JBehave, Concordian and Gauge) are often used in test automation, which is wrong!

The fact: few engineers have seen, let alone implemented, a successful Automated E2E regression testing. Check out the AgileWay Continuous Testing Grading.

Gherkin is a wrong choice for Test Automation

Don’t just take my word for it. Have a look at the following quotes from Aslak Hellesøy, the creator of Cucumber.

“Using UI testing tools together with Cucumber? Please don’t — or at least do it very sparingly.” (link)

“If all you need is a testing tool for driving a mouse and a keyboard, don’t use Cucumber. ” (link)

For more, check out my articles:

• BDD Clarified: BDD ≠ “Given-When-Then” (Gherkin)
• Why Gherkin (Cucumber, SpecFlow,…) Always Failed with UI Test Automation?

How to avoid using Gherkin in your project?

Frankly, if management and architects have already decided to adopt Gherkin for BDD test automation, it is almost impossible to change their minds. This is how office-politics works, don’t waste your time debating. But if the management asked you for opinions, and you know “Cucumber/SpecFlow/JBehave” is a road to failure, how can you convince others?

First of all, it is not easy. Because some already were hyped about using BA-written (Given-When-Then) requirements for automated tests. During my contracting/consulting (over 10 years), I failed to change their mind. I used the expression such as “Most test automation with Cucumber failed”, I even borrowed the quotes from absolute authorities such as Cucumber’s creator and DHH, but still no use. Someone would say: “It worked very well in my last project, …”. Confrontation seemed inevitable.

I changed tactics at one company, and with some degrees of success. Background information: all previous test automation attempts have failed for this large financial company in the past 12 years. The newly-joined QA director called for a meeting. I declined the meeting request (just wanted to focus on my own project, as a contractor, I was not interested in meetings beyond my daily work), but the QA director heard of me and insisted.

In this meeting, one suggested using Cucumber for test automation. A few concurred. The QA director looked around for opinions. I could sense he was going to support that. I said: “

I have been hands-on working on web test automation since 2006 with a good track record;

Authored 10 books on Test Automation and Continuous Testing;

Spoke at several international Software Testing Conferences;

Created a functional testing IDE that supports Cucumber;

Created an international award-winning CT server that supports Cucumber

But I have no confidence to maintain a suite of 50 automated E2E tests in Cucumber. Currently, my team (3 manual testers whom I was mentoring) is quite comfortable with ~120 tests in the RSpec BDD framework, which we run all tests daily as regression testing.

The reason is simple: the extra layer of test-specific parser for English will require huge maintenance effort. ”

You feel free to quote my words to support your argument, “Zhimin Zhan would be struggling with maintaining 50 cucumber E2E tests for the app under active development (change frequently). But he is very comfortable with 500+ User-Story-Level E2E tests in RSpec.”

The meeting room was silent. This time, to my surprise, no objections, but no support either. The QA director quickly wrapped up the meeting. During my stay there, to my knowledge, Cucumber (or another Gherkin framework) was not adopted. I did not receive another invite to this kind of meeting. At least, my team was left alone to test happily using RSpec.

Afterword

About one year after I left the company, one former colleague told me that Cucumber was introduced in End-to-End Test Automation, several months later, and failed, of course. Again, test automation was rarely mentioned in this company, back to Manual Testing.

Why did this advice work?

• My convincing records on Cucumber (Gherkin) and test automation/CT in general.

Because I listed my expertise in test automation quite specifically, even people, who don’t know me well, would think I have quite a lot of experience with test automation, including Cucumber.

• I emphasized “hands-on”

Many test leads are comfortable talking about Test Automation and CI/CD. However, they probably have not hands-on written one real automated test for years. This set the context.

• A very specific limit (50) of my capability number of tests.

If someone objected, they would be measured against this number, which does not sound big at all. These people (tech leads or senior engineers) generally are comfortable with talking at an abstract level. But when they will be judged hands-on against a very specific target, …

Furthermore, I emphasised running all tests daily, a real CT practice that they probably only read from books/talks.

• I outlined the pain point: maintenance.

Clearly, these meeting attendees don’t really know Cucumber/Gherkin well. Even so, they should know test maintenance (test scripts) is not easy, but it’s also where the majority of efforts are.

• My current team does automated regression testing daily.

In fact, multiple times a day, running all tests in a BuildWise CT server on my local machine. Regression Testing is a topic that all software professionals know, but few witnessed a good implementation.

• I did not object directly to the ones who suggested Cucumber.

I expressed that “I couldn’t”, and did not say “they could not”. They are more likely to accept that.

• I support BDD, but in RSpec.

Even though our interpretations of BDD are different and most of them did not understand RSpec at all, we agreed on the method of BDD.

--

Related reading:

• My eBook: “Practical Web Test Automation with Selenium WebDriver”
• My eBook: “Practical Continuous Testing: make Agile/DevOps real”
• Benefits of Test Automation and Continuous Testing series: Executives, Managers, Business Analysts, Developers, Testers and Customers.