Story: A Former Mentee Failed Test Automation with Protractor. Part 1

The price to pay for not following a proven winning formula.

By Zhimin ZhanPublished 9 months ago • Updated 9 months ago • 4 min read

Besides my own software apps, I have been implementing real test automation & Continuous Testing for a number of client projects by using the exact same formula. The test automation framework used is raw Selenium WebDriver in RSpec.

As you might have known, JavaScript has been the favourite language in recent years and is used in Protractor, TestCafe, Puppeteer, Cypress and Playwright. As a result, many automation testers (though some of them knew Ruby is better, including those whom I mentored) went for Javascript for more job opportunities or give in under the pressure. Here I have an interesting real story to share.

The Story

Many years ago, I took a tech lead role in a small start-up company for 3 months. Besides all the usual hands-on development /testing /DevOps tasks, I also mentored the team.

F was a final-year Uni student when I met him, who had already been there working as a part-time manual tester. I taught him automated testing, Ruby, using Git, Continuous Testing, and later, app development. He was very smart (I vaguely remembered he received a Uni Medal for his academic records) and learnt well during my 3-month consultation role there. By the time I left the company, I had set up the regression suite consisting of 148 Selenium RSpec tests that were running fine daily in a BuildWise CT Server with 3 build agents running on three Mac minis. F had been involved in real test automation hands-on under the proper guidance.

Below is the thank-you email he sent to me after I left.

A few months later, a former colleague met me on the street. He told me the automated regression testing was discarded as the boss wanted the team to just focus on developing new features, ignoring test failures in the CT server. Shortly after, all kinds of quality issues came up. The boss and their partner had to get involved in the manual testing, even on weekends. About a year later, the company went bankrupt.

Several years passed. In October 2019, I received F’s invitation to a get-together lunch, and I went.

During the catchup, F said that he had been working as a developer since graduation. He thanked me again for my mentoring and asked for my advice to implement test automation in this new job. I replied: “raw Selenium WebDriver in RSpec, exactly the same as what you did before”. He said: “The company’s app is developed in JavaScript”. I smiled and said: “It doesn’t matter. Functional UI testing is irrelevant to what language you use, you shall know that”.

Two years later (August 2021), F sent me this LinkedIn message. (F’s job title, according to his LinkedIn Profile, is Senior Software Engineer)

Apparently, he ignored my advice. The result turned out as exactly what I had predicted. From his short message, I can see a few problems:

TypeScript - This is wrong. They shall use a proper scripting language such as Ruby. Check out this article.
Flaky E2E Tests - It seems ‘a news’ to him, but it shouldn’t be the case. During the mentoring, I showed him many ways to improve the reliability of test execution, like some in this article: Working Automated Test ≠ Good Reliable Test.
Running Automated E2E Tests in Jenkins - This is wrong too. Jenkins is a CI server that is built for executing unit tests. They should use a CT server, which he realized (reflected in his last sentence). Check out this great presentation “Continuous Integration at Facebook”.
Scheduling E2E test execution every hour - Wrong again. The mindset behind this bad practice is to assume that test execution is repeatedly reliable. I will cover this in a separate article.

Above all, it was the bad choice of test automation framework: Protractor 😥. It should be raw Selenium WebDriver with RSpec, the winning formula he used years ago. By the way, Protractor was deprecated in 2021, I was not aware of that when I received F’s message.

Clearly, F still wanted to stick with Protractor. After failing with Jenkins (not surprisingly), he wished to use my BuildWise CT server (that he has used before under my mentoring).

My reply to him: “Yes, BuildWise supports RSpec, Cucumber, Pytest and Mocha. If your Protractor tests are in Mocha, they can be run in BuildWise. However, I will still strongly recommend RSpec.” In addition, I answered several other questions and provided some resource links.

F thanked me for that, and that was the last time I heard from him. Obviously, he did not take my advice. He changed his job a few months later (from a LinkedIn update).

Some might wonder: “Given that you two live in the same city and he knew that you would get test automation working, why didn’t his company engage you?”

Well, I can assume that there could be many reasons:

F was unable to convince the management who did not take test automation seriously.
F did not really care as it was not his company anyway.
F underestimated the effort of test automation and CT. According to his career history, he started working as a software engineer after graduation. He might vaguely remember how easily and effortlessly I accomplished real test automation. But when he started doing it, it was a different story.
F overestimated his ability in JavaScript and test automation/CT when he came to converting my winning formula.

As you can see, for any of the last two reasons, asking his company to engage me as a test automation coach might ruin his reputation in test automation in his company. Therefore, F was trying to get some quick free consultation for old-time’s sake. I did provide candid answers and resources.

In Part 2, I will share my reflections on this true story and answer questions some might have.

This article was originally published on my Medium blog on 2023-05-27.