Case Study: Stabilizing Date Test Data in a Large E2E (via UI) Test Suite

This is an abridged version of my Medium article.

One common excuse for failed E2E test automation attempts is “Test Data is outdated”. It is a poor and wrong excuse.

“Test Data is a part of Test Scripts. Why? It is included or used in the test scripts.” — Zhimin Zhan

We are just three days away from the start of the new year. Experienced software testers know that commonly date-sensitive tests are very likely to fail on those dates (known as edge cases), especially in a large suite like this one: Showcase a 500+ End-to-End (via UI) Test Suite.

Fixing Two Date-Related Test Failures for the WhenWise suite

WhenWise is a commercial service-booking app I developed based on my daughter’s idea and prototype (when she was in high school). Obviously, the nature of this app means it is quite date-sensitive.

BTW: people can book my training and coaching on WhenWise.

At 9:32 on 28th December, I triggered a regression run.

The build failed with six failing tests. After analysis, they fit into two categories:

1. The Test Credit Cards on Stripe (payment gateway) expired.

2. The membership end date in an Excel spreadsheet

This is a test loading an external membership from a spreadsheet.

The above test data issues are all external (not directly used in actual test scripts). For a 570-test suite, it is quite good.

I quickly fixed these six tests by updating the testing credit card on Stripe and the Excel file and verifying that against my local server while the build was running.

I triggered another run in BuildWise at 10:56 AM.

It passed. As a standard practice in my company, I pushed the new build to production after a PASS of the E2E Selenium Test Suite on BuildWise CT Server (see Daily Production Releases).

Astute readers might have the following observations

1. The second build time is shorter, 42 minutes vs 51 minutes

This is because I engaged one more BuildWise Agent, 5 in total. More build agents means a shorter overall execution time.

2. The failed tests in the later build seemed to get run first

Yes, this feature in BuildWise called “Dynamic Priority” provides a quicker feedback loop.

For more on Continuous Testing, executing a large number of E2E (via UI) tests frequently as regression testing, check out my book, “Practical Continuous Testing: make Agile/DevOps real”.

How do I handle the dates in test data?

Test data in E2E test automation deserves a series of articles. Here, I will just highlight the key points:

1. You need to spend considerable effort on Test Data

A common mistake that I found with automated testers is that they think the job is done (or mostly done) after getting one good run. Far from that. We need to

• optimize (for speed, readability),
• refactor (for maintainability and readability),
• stabilize test data. Ask yourself questions, “Will the test scripts still be valid if run tomorrow, or next month/year, or on special dates, e.g. 02–29, …?”

To give a feel of how serious I am on test data. I implemented the database reset (I call it The Simpsons’ Data Reset Pattern) in the WhenWise app.

To put it simply, I seed the database with pre-defined data. For example, the customer's credit card expires “Next Year”.

PaymentMethod.create(:business_id => wise_yoga.id, :name => "Visa", :card_number => "************4242", :expiry_date => Date.today.next_year)

2. Be Dynamic

Try to use dynamic dates as much as possible. Instead of send_keys("2024-01-01") , do this send_keys(Date.today.next_year.to_s) .

Astute readers might find the date operations in the above steps easy to read. Yes, thanks to the great ActiveSupport package in the wonderful Ruby language, we can use syntax such as “2.years.from_now”.

3. Run the whole suite often

No matter how carefully we plan for the test data, it will have some issues now and then. (We are human beings). The simple and logical solution is to run the whole suite often. Preferably twice a day when changes are made or at least once per week, even without app changes. In the latter case, the date changed.

By adopting this approach, we can identify date-related issues incrementally, making them easier to address.

I’ve observed numerous unsuccessful attempts at test automation, particularly in projects where the test suite hasn’t been executed for several months. When someone finally brings it up and initiates a run, many test failures occur, leading to a loss of confidence in rectifying the situation. Ultimately, the entire effort ends up being wasted. Two common excuses by incompetent ‘automated testers”: “The app changed” and “The test data is no longer valid”.