Comparing the Cost of End-to-End Test Execution in CT Servers: Cypress Cloud and BuildWise

One user’s feedback on my article “Comments on the claims of ‘7 ways Cypress is different’. All False, Wrong or Lie”: “Cypress Cloud is good”. Oh well, I haven’t used it. For failed Cypress test automation attempts I rescued, they were in a complete mess, previously planned Continuous Testing never occurred. On rescue missions, I usually got some converted (from Cypress to the great raw Selenium WebDriver + RSpec) running in the BuildWise CT server on Day 1, then every day onwards.

Because lacking Cypress Cloud experience, I can’t comment on its functionality. However, the cost comparison is fine based on the public information on the Cypress Cloud website.

I have nearly 1 million (raw Selenium WebDriver + RSpec) test executions for my two highly dynamic web apps: ClinicWise and WhenWise, in BuildWise CT Server over the last 10 years.

By the way, this might remove the doubt of Selenium for some Cypress testers, who fall into the big-fat lies of “Selenium is flaky”. Remind you, Cypress, so so-called “Selenium Killer” is dying; Selenium WebDriver is still strong, and loved by real test automation engineers.

If a tester understands Selenium WebDriver, he will refrain from making frivolous remarks like “Selenium is flaky”. If someone does, grow up. Selenium WebDriver is the most reliable automation framework, far ahead of any others.

1. W3C Stanard. (how do you trust product quality? check ISO 9001 standard, right?)

2. Created by world-class engineers. (mostly from Google and Facebook)

3. Supported by all major browser vendors. (this means a lot)

Check out my article series: Flaky Tests in Test Automation Clarified

As is the case with any business decision, the cost must be taken into careful consideration. In this article, I will compare the cost of running the size of my two test suites BuildWise CT Server and, if, Cypress Cloud.

Some readers might wonder, “Why don’t you compare your solution against Jenkins, Bamboo or TeamCity, popular ones?”. Those servers are Continuous Integration (CI) servers, not optimized for executing end-to-end testing.

If hypothetically, I chose Cypress Cloud to run my test suites (I am highly doubtful Cypress Cloud can produce reliable results, anyway), even the lowest “TEAM” plan would cost me $799/year, that’s US$7990 for 10 years. Most likely, I would need the recommended “BUSINESS” plan, which would cost me US$31,990 for ten years. Please note, this is the starting price. As we know, businesses usually often end up paying a fair bit more than that.

Some might say, $32K is really not much for a software business. I agree, do you want to know how much my Continuous End-to-End Testing Solution cost?

1. the award-winning BuildWise Server is free and open-source. which means you don’t have any restrictions, unlimited projects, unlimited users, …

2. The build agent cost $30/per month, 25% discount for renewals. I used 6 build agents to execute tests.

3. On-premise Hardware (for build agents) cost is neglectable. Besides a server (I used a Mac Mini), test executions are in virtual machines of several MacMinis, which are still running strong. In addition, you have the option of utilizing refurbished Windows machines as build agent machines, typically priced at around $100 each.

As you can see, the cost is mainly on the number of BuildWise Agents you use.

For large test suites consisting of 500+ raw Selenium WebDriver tests, as the WhenWise suite showed here, I typically use 5 buildwise agents. Very few software companies have the capability to maintain a suite of 500+ user-story level end-to-end UI tests (Level 4 of AgileWay Continuous Testing Grading) as regression testing. Usually, I recommend three build agents, which is sufficient for most.

The total cost (with 3 build agents) is under $8370 for 10 years, plus maybe neglectable hardware cost. Please remember, different from Cypress Cloud, you have full control of this Continuous Testing lab, without any limitations (such as data retention days), security concerns (such as leaking sensitive data outside the company firewall), the existence of the company (like Cypress.IO is dying), and other costs.

For teams on a tight budget, they can even run full-featured BuildWise Agents in free mode (up to 40 minutes, like Zoom’s limitation, a 3-second restart to continue), and then it is free in terms of software cost.

Competent software engineers can extend the BuildWise server. For example, it only took me one day to add support executing C# tests (previously, only support tests in scripting language frameworks, such as RSpec, Pytest, Cucumber, and Mocha). This is for my daughter, who is currently authoring a book for Apress publisher. If you are interested in running C# tests in BuildWise, stay tuned for her next week’s article.

For ambitious software engineers in a large organization, you can even develop your own BuildWise-Server-Compatible agents in any language, such as Java or JS. The API is defined in the BuildWise Server, which is free and open-source. Doing so gives you a completely free solution, and you can even sell it.

Some desperate Cypress defenders would say, “I think Cypress Cloud is better than BuildWise”. Maybe or may not, but as I said earlier, I never used and didn’t want to touch Cypress Cloud. Three facts about BuildWise:

• BuildWise Won Ruby International Award in 2018, assessed by the software legend, Matz
• BuildWise Server is free and open-source. For competent software engineers, it is hackable (in a good way).
• I have been using BuildWise for over 10 years, and I see myself continue doing so in many years to come.

However, dear Cypress testers, you may have to bid farewell to Cypress Cloud, as Cypress.IO is facing an uncertain future. In the next article, I would like to offer my suggestions to Cypress testers who are concerned and open to receiving my advice. After all, from a technical perspective, I predicted the current Cypress situation years ago.