“Daily Production Releases” Clarified, Part 2

This article is one of the “IT Terminology Clarified” series.

This is abridged version of the original article on my Medium blog, 2022-11-10.

Continue from Part 1.

4. Automated End-to-end (via GUI) regression testing is the enabler of “Daily Production Releases”

Let me ask you a question first: “How long does a round of full regression testing take in your project?”.

If the answer is “1 week”, then surely “daily production releases” is not possible.

Averagely speaking, a properly-managed software team needs to perform 2–4 full regression testing rounds (and one pass) for a production release.

The answer is not “8 hours” either, shall be much shorter than that. Why? you cannot assume that the result of regression testing is always “ALL PASS”. In fact, on the opposite, good regression testing detects regression errors. After developers fixed the identified regression defects, there would be another build, thus, another round of regression testing. Then, possibly a new set of regression errors! , … I think you get the picture.

Therefore, a round of comprehensive regression testing shall only take an hour or less to achieve “Daily Production Releases”.

• Test Automation

Manual testing simply is not quick and reliable enough.

• Comprehensive

Ideally, every user story is covered by automated E2E tests (Done, Done in Agile).

• End-to-End Testing (via GUI)

Unit testing and integration testing are not enough. A simple comma in JavaScript or CSS might cause web pages to fail to load. You do need end-to-end testing via (GUI).

Some might wonder how “End-to-End Testing” fits in SDLC and release, besides the above Facebook representation, read the below (from LinkedIn).

LinkedIn’s test automation enabled frequent releases.

“Newly-added code is subjected to an elaborate series of automated tests designed to weed out any bugs. Once the code passes the tests it is merged into trunk and cataloged in a system that shows managers what features are ready to go live on the site or in new versions of LinkedIn’s apps.”

— The Software Revolution Behind LinkedIn’s Gushing Profits (2013)

5. How “Daily Production Releases” is possible?

Yes, it is. As a matter of fact, once you have experienced one, you will know that is how natural software development shall be.

The technical how is comprehensive Automated End-to-End (via UI) regression testing, running multiple times a day in a continuous form. For more, check out How to Implement Real Automated Regression Testing?

A few words on the test execution time. The majority of software teams attempted test automation, failed and gave it up quietly. The reason: they were unable to maintain when a test suite grew to ~50 test cases, while the application changes constantly. Growing-test-count means a lot more maintenance effort, why? A run will take a long time. The solution: Run the whole end-to-end test suite in a Continuous Testing sever such as BuildWise, please note, not Jenkins or Bamboo, which are CI servers.

Here is a recent test report of my WhenWise app, its automated end-to-end regression suite consists of 558 raw Selenium WebDriver tests.

As you see, a run of this test suite (over 25000 steps) takes about 200 minutes on a single machine. By running the tests in parallel on 5 build agents, the total execution time is reduced to 46 minutes. For more on Continuous Testing, check out my book “Practical Continuous Testing: make Agile/DevOps real”.

6. “Daily Production Releases” means a complete change to the software development process.

Most IT engineers work on 3+ month release schedules. Now, somehow, magically, your team is doing real ‘daily production releases’. Consider the following activities you were used to:

• Daily stand-up meeting
• Spring planning
• Velocity Chart
• Retrospectives
• The work of the Release Manager
• Defect Tracking process
• Regression Testing process
• …

Surely, there will be changes, right?

Wired used “completely overhauled” to describe the changes made in LinkedIn.

“It was Scott and his team of programmers who completely overhauled how LinkedIn develops and ships new updates to its website and apps, taking a system that required a full month to release new features and turning it into one that pushes out updates multiple times per day”

— The Software Revolution Behind LinkedIn’s Gushing Profits (2013)

7. Most CIOs, Managers, Tech leads, and Software Engineers chickened out in front of “Daily production releases”

Senior executives like to talk about how much they desire “Daily production releases”. Warning: don’t take that seriously. In my experience, the reality is often the opposite: those people fear “daily production releases”. “Release often” is OK, but how often can be interpreted differently, but not “Release daily”.

How come? There is an ancient Chinese idiom that describes this mindset: Lord Ye Loves Dragon. I also shared one story in that article.

For readers who are still doubtful? Considering this scenario. If all wishes become true, your company suddenly is doing production releases twice a day, what will your CIO, senior VP, programme director/managers, architects and principal software engineers do every day? Surely, it is hard to justify several project development meetings every day. Generally speaking, human beings fear changes, especially ones that bring unknown impacts. Talking about, not doing, is safer.

---

Further reading:

• My eBook: “Practical Web Test Automation with Selenium WebDriver”

• Benefits of E2E Test Automation and Continuous Testing series: Executives, Managers, Business Analysts, Developers, Testers and Customers.
• Test Automation Camel, a metaphor that explains why most test automation attempts failed?

• Set up, Develop Automated UI tests and Run them in a CT server on your First day at work