Blue-green vs. canary: how to deploy without taking down production

⏱️ Lectura: 13 min

A team deploys new code at three in the afternoon, and within seconds the site starts returning 500 errors to thousands of users. Rolling back manually takes ten panicked minutes while complaints pile up on social media.

📑 En este artículo

1. TL;DR
2. What is canary deployment (and what is blue-green)
3. How a canary deployment works in detail
4. Practical examples: from a simple Service to a canary Rollout
5. Getting started: installing and running your first canary deployment
6. Real-world use cases
7. Common mistakes and best practices
8. Comparison: rolling update, blue-green, canary, and feature flags
9. Going deeper: traffic mirroring and statistical analysis
10. Frequently Asked Questions
    1. Are blue-green deployment and canary deployment mutually exclusive?
    2. Do I need a service mesh like Istio to do canary deployment?
    3. What happens to the database during a blue-green deployment?
    4. How much traffic should a canary receive at the start?
    5. What open source tools implement canary deployment on Kubernetes?
    6. Is Kubernetes’ rolling update no longer useful then?
11. References

Canary deployment and blue-green deployment exist precisely to avoid that scenario. Both let you bring new code into production without risking all real traffic at once, and roll back in seconds if something goes wrong. The difference lies in how they move that traffic: blue-green switches everything at once between two identical environments, while a canary deployment shifts it gradually and measures the impact before completing it.

TL;DR

• You’ll understand the real difference between blue-green deployment and canary deployment, and when to use each one.
• You’ll be able to switch traffic in a blue-green deployment with a simple Kubernetes selector.
• You’ll be able to write an Argo Rollouts Rollout that deploys a canary with controlled traffic increments.
• You’ll understand why an incompatible schema migration breaks a zero-downtime deployment.
• You’ll be able to compare rolling update, blue-green, canary, and feature flags in a table and choose the right one for your case.
• You’ll know which metrics to watch to automate a rollback before a canary damages production.

What is canary deployment (and what is blue-green)

Think of a blue-green deployment as two identical theater stages: one with the current play (blue) and another with the new play already set up (green). The audience, that is, real traffic, is seated facing the blue stage. When the green cast finishes rehearsals, a single switch turns off the lights on blue and turns them on for green. The audience never notices the stage change; they just see the new play from the next second on.

A canary deployment works differently. Instead of moving the whole audience at once, you let a small group (say 10% of the spectators) watch the new show while the rest keep watching the old one. If that group reports problems, something equivalent to a rise in error rate or latency, you cancel the new show before the rest of the audience sees it. The name comes from the canaries miners used to carry underground: if the bird passed out from toxic gas, the rest of the crew knew to get out before the danger reached everyone.

Both techniques solve the same underlying problem: the moment a new software version touches real traffic for the first time is, almost always, the riskiest instant in a system’s lifecycle. No staging environment exactly reproduces the volume, data variety, and behavior of real users.

How a canary deployment works in detail

In blue-green, the mechanics are simple: there are two complete environments (compute, and sometimes database), and a router, load balancer, DNS record, or Kubernetes selector points to just one of the two. The deployment happens in the inactive environment, smoke tests run against it without exposing it to the public, and then the traffic switch is atomic: it goes from 0% to 100% in a single operation. The rollback is symmetric: point the router back to the previous environment.

Canary deployment needs a different mechanism: weighted routing. An ingress controller, a service mesh (Istio, Linkerd), or a tool like Argo Rollouts splits traffic between two groups of pods running different versions of the same service, according to a configured percentage. That percentage rises in defined steps, almost always with a pause between each step so an automated analysis system can compare the canary’s metrics against the stable group’s before moving to the next step.

flowchart LR
    U["Users"] --> LB["Load Balancer"]
    LB -->|"active traffic"| B["Blue Environment (current version)"]
    LB -.->|"standby"| G["Green Environment (new version)"]
    subgraph Production
    B
    G
    end

The diagram above shows the state before the swap: the load balancer sends all traffic to blue while green waits, already deployed and tested, ready to receive 100% in a single operation.

Practical examples: from a simple Service to a canary Rollout

The simplest example of blue-green in Kubernetes needs no external tools: a Service whose selector points to a version label is enough.

apiVersion: v1
kind: Service
metadata:
  name: checkout-service
spec:
  selector:
    app: checkout
    version: blue
  ports:
    - port: 80
      targetPort: 8080

This Service routes traffic only to pods labeled version: blue. When the green environment passes smoke tests, the only change needed is to edit that selector:

kubectl patch service checkout-service \n  -p '{"spec":{"selector":{"version":"green"}}}'

That command switches the traffic all at once: pods labeled version: green start receiving requests as soon as Kubernetes updates the Service’s internal endpoints. There is no mixing period between versions.

A real canary deployment needs more granularity than a binary selector. Argo Rollouts replaces the standard Deployment resource with a Rollout that defines explicit traffic steps:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout-rollout
spec:
  replicas: 10
  strategy:
    canary:
      steps:
        - setWeight: 10
        - pause: {duration: 300}
        - setWeight: 50
        - pause: {duration: 300}
        - setWeight: 100
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      containers:
        - name: checkout
          image: registry.example.com/checkout:2.4.0

Each setWeight defines the percentage of traffic that sees the new version; each pause halts progress for that number of seconds to give time to observe metrics before continuing. With 10 replicas, a setWeight: 10 corresponds to roughly one canary pod and nine from the stable group.

sequenceDiagram
    participant O as Operator
    participant R as Argo Rollouts
    participant C as Canary Pods
    participant S as Stable Pods
    O->>R: applies new version
    R->>C: routes 10% of traffic
    R->>S: keeps 90% of traffic
    Note over R,C: analyzes error and latency metrics
    R->>C: increases to 50% of traffic
    R->>C: promotes to 100% of traffic
    R->>S: retires previous version

Getting started: installing and running your first canary deployment

These steps assume a test Kubernetes cluster (kind or minikube work fine) and kubectl configured.

# 1. Install the Argo Rollouts controller
kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts \n  -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

# 2. Install the kubectl plugin
curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64
chmod +x kubectl-argo-rollouts-linux-amd64
sudo mv kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts

# 3. Apply the Rollout defined above
kubectl apply -f checkout-rollout.yaml

# 4. Trigger a new release by changing the image
kubectl argo-rollouts set image checkout-rollout \n  checkout=registry.example.com/checkout:2.5.0

# 5. Watch the rollout progress live
kubectl argo-rollouts get rollout checkout-rollout --watch

To confirm the canary is actually active and not at 0%, run kubectl argo-rollouts status checkout-rollout: the current step and traffic weight fields show the exact percentage the canary is receiving at that moment, without relying on what the static YAML configuration says.

💡 Tip: before manually promoting a paused step, run kubectl argo-rollouts get rollout checkout-rollout and check the canary vs. stable replica count: if it doesn’t match the configured weight, something in the automated analysis is holding it back.

Real-world use cases

An API version migration where a contract change could break old clients is a typical canary deployment case: exposing the new contract to 5% of traffic reveals, before breaking everyone, whether some integrated client stops working.

Switching payment providers in a checkout flow is a case where blue-green usually beats canary: if the new provider fails in the first minute, an instant rollback avoids leaving shopping carts half-processed with two providers mixed together.

Retraining a recommendation or scoring model also lends itself to canary with automated analysis: comparing the click-through or conversion rate of the new model against the stable model, with real traffic, before replacing it completely.

An infrastructure migration, like moving a service from one datacenter region to another, is usually solved with full blue-green: both environments run in parallel during the migration, and the final DNS or Service swap avoids any downtime window.

Common mistakes and best practices

Stateful sessions are blue-green’s most common trap: if a user has their session stored in a blue pod’s memory and the swap redirects them to green, they lose the session instantly. The solution is to externalize sessions (Redis, for example) instead of keeping them in the process’s local memory.

Backward-incompatible schema migrations break both canary and blue-green. If the new version runs a migration that drops a column the stable version still reads, the stable version starts failing while both coexist. The golden rule: every schema migration must remain compatible with the previous version throughout the entire deployment window.

Keeping two complete environments in blue-green doubles compute spending for as long as the deployment lasts. For large services this isn’t free, and it’s one of the reasons many teams prefer canary: it never runs 100% duplicated capacity, only the percentage of the current step.

A canary deployment without automated thresholds is purely decorative. Watching a dashboard for two minutes and deciding “it looks fine” doesn’t replace a configured threshold like Flagger’s: without that automation, the rollback decision rests on a human who might not catch a small degradation in time.

Finally, a CDN or browser cache might keep serving static assets from the old version after the swap, mixing new HTML with old JavaScript. The standard practice is to version static files with a hash in the filename.

⚠️ Watch out: blue-green gives you an instant rollback, but it doesn’t protect against business logic bugs that only show up with real traffic, because 100% of traffic is already exposed the instant the swap happens. That’s where canary deployment has a real advantage: it exposes the risk to a controlled fraction before committing all traffic.

Comparison: rolling update, blue-green, canary, and feature flags

Option	When to use it	Advantage	Limitation
Rolling update	Low-risk changes, with no expected incompatibility	Native to Kubernetes, no extra tools	No instant rollback; mixes versions during the rollout
Blue-green deployment	Large changes where rollback must be immediate	Rollback in seconds, no version mixing	Doubles infrastructure cost for the duration of the deployment
Canary deployment	Uncertain impact on real users that can be measured	Exposes risk to a controlled fraction; automated rollback based on metrics	Requires analysis infrastructure (Prometheus, service mesh)
Feature flags	Turning functionality on or off without redeploying	Granular control per user or segment	Accumulates technical debt if old flags aren’t cleaned up

Going deeper: traffic mirroring and statistical analysis

The next level of sophistication in canary deployment is traffic mirroring (or shadow traffic): a service mesh like Istio duplicates every real request and sends a copy to the canary, but discards the canary’s response before it reaches the user. This allows testing the new version against real production traffic with zero risk, because the user never sees the canary’s response.

Tools like Flagger go a step beyond a simple fixed threshold: they query Prometheus metrics at each interval and use the comparison between the canary group and the stable group to decide whether to promote, pause, or roll back. A typical configuration defines a success-rate and latency threshold per metric:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: checkout
spec:
  analysis:
    interval: 1m
    threshold: 5
    maxWeight: 50
    stepWeight: 10
    metrics:
      - name: request-success-rate
        thresholdRange:
          min: 99
        interval: 1m
      - name: request-duration
        thresholdRange:
          max: 500
        interval: 1m

With this configuration, Flagger queries Prometheus every minute. If the success rate drops below the min: 99 threshold or request duration exceeds max: 500 (in milliseconds), Flagger automatically reverts the canary’s weight to 0, without waiting for human intervention.

An important nuance with small samples: if the canary receives only 5% of traffic on a low-volume service, a few slow requests can trigger a false alert. That’s why the threshold in Flagger’s configuration isn’t a simple binary cutoff, but a consecutive-failure counter: it only rolls back after several consecutive out-of-range measurements, not on a single noisy data point.

flowchart TD
    A["Degraded metrics detected"] --> B{"Error rate > threshold?"}
    B -->|"yes"| C["Automatic rollback to 0% canary"]
    B -->|"no"| D["Continue increasing traffic"]
    C --> E["Alert the team"]
    D --> F["Promotion to 100%"]

💭 Key point: automated canary deployment doesn’t eliminate deployment risk, it turns it into a measurable experiment: each traffic step is a question (“does this version behave the same as the previous one?”) with a quantifiable answer before committing the rest of the traffic.

📖 Summary on Telegram: View summary

Your next step: spin up a local cluster with kind or minikube, install Argo Rollouts, and convert a sample Deployment into a canary Rollout with two setWeight steps to watch the traffic split live.

Frequently Asked Questions

Are blue-green deployment and canary deployment mutually exclusive?

No. Many teams use blue-green for the final switch between complete environments and canary deployment within the new environment to gradually expose traffic before the full switch.

Do I need a service mesh like Istio to do canary deployment?

It’s not mandatory. Tools like Argo Rollouts or the NGINX ingress controller support percentage-based traffic splitting without a full service mesh, though a mesh gives more request-level control.

What happens to the database during a blue-green deployment?

Generally both environments share the same database. That’s why every schema migration must be backward compatible during the transition, or you need a blue-green strategy at the database level too.

How much traffic should a canary receive at the start?

It depends on the service’s total volume: the lower the traffic, the higher the initial percentage needs to be to gather enough data and detect a real degradation quickly.

What open source tools implement canary deployment on Kubernetes?

Argo Rollouts and Flagger are the two most widely used options; both integrate with Prometheus to automate metrics analysis and rollback.

Is Kubernetes’ rolling update no longer useful then?

It’s still the right choice for low-risk changes where you don’t need to measure the impact on real traffic before completing the deployment.

References

• Martin Fowler, BlueGreenDeployment: the original description of the blue-green pattern.
• Kubernetes docs, Deployments: native rolling update strategy and its configuration.
• Argo Rollouts, official documentation: the Rollout resource, canary steps, and automated analysis.
• Flagger, official documentation: progressive canary analysis with Prometheus.
• Istio, Traffic Management: weighted routing and traffic mirroring at the service mesh level.

📱 Like this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day.

Imagen destacada: Foto de Fotis Fotopoulos en Unsplash