01 logo

“Daily Production Releases” Clarified, Part 2

“Daily Production Releases” is mandatory for real Agile/DevOps.

By Zhimin ZhanPublished 2 months ago 4 min read

This article is one of the “IT Terminology Clarified” series.

This is abridged version of the original article on my Medium blog, 2022-11-10.

Continue from Part 1.

4. Automated End-to-end (via GUI) regression testing is the enabler of “Daily Production Releases”

Let me ask you a question first: “How long does a round of full regression testing take in your project?”.

If the answer is “1 week”, then surely “daily production releases” is not possible.

Averagely speaking, a properly-managed software team needs to perform 2–4 full regression testing rounds (and one pass) for a production release.

The answer is not “8 hours” either, shall be much shorter than that. Why? you cannot assume that the result of regression testing is always “ALL PASS”. In fact, on the opposite, good regression testing detects regression errors. After developers fixed the identified regression defects, there would be another build, thus, another round of regression testing. Then, possibly a new set of regression errors! , … I think you get the picture.

Therefore, a round of comprehensive regression testing shall only take an hour or less to achieve “Daily Production Releases”.

  • Test Automation

Manual testing simply is not quick and reliable enough.

  • Comprehensive

Ideally, every user story is covered by automated E2E tests (Done, Done in Agile).

  • End-to-End Testing (via GUI)

Unit testing and integration testing are not enough. A simple comma in JavaScript or CSS might cause web pages to fail to load. You do need end-to-end testing via (GUI).

Facebook’s Testing Pyramid, because it uses WebDriver, so calls “WebDriver” tier. Check out “Continuous Integration at Facebook” video on Youtube.\

Some might wonder how “End-to-End Testing” fits in SDLC and release, besides the above Facebook representation, read the below (from LinkedIn).

LinkedIn’s test automation enabled frequent releases.

“Newly-added code is subjected to an elaborate series of automated tests designed to weed out any bugs. Once the code passes the tests it is merged into trunk and cataloged in a system that shows managers what features are ready to go live on the site or in new versions of LinkedIn’s apps.”

The Software Revolution Behind LinkedIn’s Gushing Profits (2013)

5. How “Daily Production Releases” is possible?

Yes, it is. As a matter of fact, once you have experienced one, you will know that is how natural software development shall be.

The technical how is comprehensive Automated End-to-End (via UI) regression testing, running multiple times a day in a continuous form. For more, check out How to Implement Real Automated Regression Testing?

A few words on the test execution time. The majority of software teams attempted test automation, failed and gave it up quietly. The reason: they were unable to maintain when a test suite grew to ~50 test cases, while the application changes constantly. Growing-test-count means a lot more maintenance effort, why? A run will take a long time. The solution: Run the whole end-to-end test suite in a Continuous Testing sever such as BuildWise, please note, not Jenkins or Bamboo, which are CI servers.

Here is a recent test report of my WhenWise app, its automated end-to-end regression suite consists of 558 raw Selenium WebDriver tests.

As you see, a run of this test suite (over 25000 steps) takes about 200 minutes on a single machine. By running the tests in parallel on 5 build agents, the total execution time is reduced to 46 minutes. For more on Continuous Testing, check out my book “Practical Continuous Testing: make Agile/DevOps real”.

6. “Daily Production Releases” means a complete change to the software development process.

Most IT engineers work on 3+ month release schedules. Now, somehow, magically, your team is doing real ‘daily production releases’. Consider the following activities you were used to:

  • Daily stand-up meeting
  • Spring planning
  • Velocity Chart
  • Retrospectives
  • The work of the Release Manager
  • Defect Tracking process
  • Regression Testing process

Surely, there will be changes, right?

Wired used “completely overhauled” to describe the changes made in LinkedIn.

“It was Scott and his team of programmers who completely overhauled how LinkedIn develops and ships new updates to its website and apps, taking a system that required a full month to release new features and turning it into one that pushes out updates multiple times per day”

The Software Revolution Behind LinkedIn’s Gushing Profits (2013)

7. Most CIOs, Managers, Tech leads, and Software Engineers chickened out in front of “Daily production releases”

Senior executives like to talk about how much they desire “Daily production releases”. Warning: don’t take that seriously. In my experience, the reality is often the opposite: those people fear “daily production releases”. “Release often” is OK, but how often can be interpreted differently, but not “Release daily”.

How come? There is an ancient Chinese idiom that describes this mindset: Lord Ye Loves Dragon. I also shared one story in that article.

For readers who are still doubtful? Considering this scenario. If all wishes become true, your company suddenly is doing production releases twice a day, what will your CIO, senior VP, programme director/managers, architects and principal software engineers do every day? Surely, it is hard to justify several project development meetings every day. Generally speaking, human beings fear changes, especially ones that bring unknown impacts. Talking about, not doing, is safer.


Further reading:

how to

About the Creator

Zhimin Zhan

Test automation & CT coach, author, speaker and award-winning software developer.

A top writer on Test Automation, with 150+ articles featured in leading software testing newsletters.

My Most Viewed Articles on Vocal.

Reader insights

Be the first to share your insights about this piece.

How does it work?

Add your insights


There are no comments for this story

Be the first to respond and start the conversation.

Sign in to comment

    Find us on social media

    Miscellaneous links

    • Explore
    • Contact
    • Privacy Policy
    • Terms of Use
    • Support

    © 2023 Creatd, Inc. All Rights Reserved.