Why I Don’t Use Selenium-Grid?

The abridged version of my article on Medium (2023-09-14)

I am a big fan of Selenium WebDriver. I have been practising/coaching web test automation using raw Selenium WebDriver since 2011 (yes, when it was first released). Also, I published six books on Selenium, including “Practical Web Test Automation with Selenium WebDriver” and the Selenium WebDriver Recipes series.

But I don’t use Selenium Grid.

I attempted Selenium Grid several times and found it not useful in practice (OK for demo). The Selenium team also seemed to realise this, Selenium Grid v4 is a total rewrite. It is better, but still, I don’t think it is up to the level I expected.

People often use it for Cross-Browser Testing.

From my experience, many projects use Selenium-Grid just for cross-browser testing. That is not surprising, even in Selenium’s introduction of Grid “…, multiple browsers and operating system combination”. Some engineers consider Selenium Grid, a free competitor to SauceLabs and Browserstack.

However, cross-browser testing has become largely unnecessary, as Chromium-based browser (e.g. Chrome and Edge) dominates the browser market. As a result, fewer and fewer people mention cross-browser testing.

We shall use Selenium, what it does best, E2E (via UI) Functional Testing, preferably running the test suite often as regression testing. In my view, Selenium Grid is not up to that challenging task.

• Complex

For every selenium grid node, you need to configure the node either by specifying command line parameters or a JSON file. It is quite complex.

• Very Limited Control

Selenium-Grid comes with a web-accessible console (the UI was very poor before v4), in my view, very basic (even with Selenium Grid 4) in terms of viewing test execution status (such as output and history) and control test executions (such as cancellation and reassignment).

• Lack of feedback

UI tests take time to execute, more tests means longer execution time. Selenium Grid’s distribution model is to reduce that. Apart from the raw execution time, there is also the feedback time. The team would like to see the test results as soon as a test execution finishes on one node. Even better, when we pass the whole test suite to the hub, it will ‘intelligently’ run new or last failed tests first. Selenium Grid, in my view, falls short of this.

• Lack of Rerun

In a perfect world, all tests execute as expected every single time. But in reality, there are so many factors that could affect the test execution:

– test statements didn’t wait long enough for AJAX requests to complete (server on load)

– browser crashes (rarely, but it happens)

– node runs out of disk space

– virus scanning process started in the background

– windows self-installed an update

In all the above cases, re-assign failed tests (i.e. rerun) to another node could save a potential good build.

• Test Scripts need to be aware of Hub.

driver = Selenium::WebDriver.for :remote, :url => "http://192.168.1.228:4444", :capabilities => :chrome)

This is an anti-pattern in CI/CD. End-to-end test scripts should be totally independent of the supporting infrastructure.

The dilemma of Selenium Grid

The Selenium team understands the importance of parallel execution for end-to-end (UI) tests. It was natural for them to come up with Client-Server architecture, basically, the same as Appium.

Usually, a CI solution has been decided by tech architects at software companies, such as Jenkins, Bamboo, TeamCity and Azure CI/CD. Most of these CI/CD solutions offer licensed build agents (the equivalent of a node in Grid).

Here are the dilemmas:

• CI servers, e.g. Jenkins, can be configured to run Selenium tests, While Selenium Grid does NOT show benefits except cross-browser testing.
• Selenium Grid did not have a good reputation.
• CI vendors are not keen to develop better support for Selenium.

So, the Selenium Grid is, in fact, has been somewhat damaging Selenium’s reputation. This explains somewhat why several bad test automation tools, such as Cypress, launched “its attack on Selenium”. By the way, Cypress.io is dying.

Solution: A Continuous Testing Server like Facebook’s Sandcastle and BuildWise

Automated UI tests are known for

• taking a long time to execute
• brittle, a minor code/infrastructure change can break many tests.

Therefore, a traditional CI server is not suitable. You shall use a proper Continuous Testing server.

For more on this topic, check out:

• a Great CI/DevOps Presentation: “Continuous Integration at Facebook”
• my eBook: “Practical Continuous Testing: make Agile/DevOps real”