New breed: Canaries that test cloud-based software

cloud-based software

Makers of cloud-based software have applied the concept of the miner’s canary to testing processes. Credit: MMMM

The always-on cloud-based software delivery systems of today have accelerated the rate of innovation on products used by enterprises and consumers alike. Software service providers can push updates to add new features almost as soon as developers have finished fine-tuning the code.

This new way of doing things certainly beats the old days when software was shipped with instructions for the customer to download and install. The business model of yesteryear did, however, give the buyer complete control over when to deploy the new software — usually on weekends and overnights for enterprises, which try to limit disruptions as much as possible.

Hence, today’s service providers must find ways to offer customers the software they expect without having to resort to planned outages for updates. Having customers spread across many time zones, for instance, makes it impossible to find an ideal outage time for all.

Canaries to the rescue

This is where canaries come in handy. Yes, modern IT has appropriated the concept of the miner’s canary for help in describing a practice that’s been proven effective for delivering software to groups of customers via the cloud.

It involves creating a new-release production environment — one that is completely separate from the baseline environment — for testing the latest version with a limited number of users, maybe as few as 1 percent. Mike Kavis, a principal architect at Cloud Technology Partners, explained how “canary releases” work in a recent blog post for The Virtualization Practice.

“Metrics are gathered and compared against baseline metrics to monitor the behavior of the new release,” Kavis explained. “Once the new release is deemed certified and stable, the remaining sessions are routed to the new environment and the baseline environment is taken off line but not deleted, just in case a rollback is needed later on.”

Kavis said he has seen organizations fill the production environment pool with the following kinds of users:

  • Internal acceptance testers
  • Beta test groups
  • Small percentage of real users selected at random
  • Specific demographic of users, based on location or customer segment

Netflix canaries

Developers and operations engineers at the video streaming service Netflix rely on what they call canary clusters to assess new code before it’s rolled out to most users. The high frequency of these deployments led Netflix to automate the process using a program that compares more than 1,000 metrics between the baseline and the canary code. It then generates a confidence store that gives Netflix a sense of how well the code will perform in wide release.

Ben Shumas, a writer for the Netflix tech blog, touted the benefits in a recent post.

“By developing comprehensive deployment automation and exposing feedback about the pipeline and code flowing through it, we’ve been able to deploy more easily and address problems earlier in the deploy cycle.”

Nick Clunn
Nick Clunn is a journalist covering the tech beat and an adjunct professor at Montclair State University. He lives in New Jersey, where he had worked as a staff writer for several leading daily newspapers and websites.
Nick Clunn
Nick Clunn
Tags: Cloud Computing,Technology
  • anna

    Too bad that’s a parakeet and not a canary.

  • anna

    Again- parakeet, not a canary.