How Blue-Green Deployments Actually Work

Blue-green deployment is often presented as the clean, grown-up way to release software. You keep the old version on one side, the new version on the other, flip traffic when ready, and roll back instantly if something goes wrong.

That description is not false. It is incomplete in exactly the places that decide whether the release is calm or expensive and chaotic.

A blue-green deployment is not primarily a build technique. It is not a CI pipeline pattern. It is not a synonym for canarying, rolling updates, or feature flags. It is a traffic cutover discipline. Two production-capable environments exist at the same time, one environment serves user traffic, the other is prepared in parallel, and some control point changes which environment is considered live.

Once you describe it that way, the hard parts become obvious. Where is the control point? Does it switch new connections only, or every request? How long do old connections survive? What happens to background workers, scheduled jobs, caches, and queues? Are both colours talking to the same database? If they are, what makes rollback real rather than theatrical? When green sends a payment request, an email, or an inventory reservation into the outside world, what exactly would you be rolling back to?

Those questions are why experienced teams do not talk about blue-green deployment as if it were one atomic action. They treat it as a coordinated set of state transitions across routing, compute, storage, workers, and observability.

This article walks through those transitions in detail. We will look at where the switch usually lives, why DNS-based blue-green is often slower than people expect, how draining actually works for HTTP and long-lived sessions, why additive database migrations matter more than the router command, how to keep background workers from going double-live, and why the real rollback window often closes much earlier than the dashboard suggests.

1. Blue-green deployment is a traffic switch between two production candidates

The cleanest definition I know is this: blue-green deployment means running two release candidates at production fidelity and changing which one receives live traffic.

The important phrase is production fidelity. Green is not a staging host with smaller limits and a different secret set. It is meant to be a genuine production candidate. If blue can handle 2,000 requests per second, terminate real TLS certificates, reach the payment provider, and emit production telemetry, green should be able to do the same before cutover.

That is what makes blue-green different from a basic rolling update.

In a rolling deployment, you mutate the live pool in place. One old instance goes away, one new instance appears, and the percentage mix changes gradually. There is no clean old world and clean new world. The fleet is mixed for some period.

In blue-green, you try to preserve a clean separation:

Before cutover:
  blue  -> serving user traffic
  green -> prepared, probed, not yet live
 
After cutover:
  blue  -> draining or parked
  green -> serving user traffic

That separation buys a few things.

First, it gives you a stable comparison point. If latency rises after cutover, you have an untouched blue environment that can still answer the question, "was this new behaviour introduced by the release or by something shared underneath both colours?"

Second, it gives you a simpler operational story for the application tier. You are not trying to reason about 17 percent of pods being new and 83 percent being old while sticky sessions pin some users to one group and some to the other. The intended model is binary: live and standby.

Third, it shortens one class of rollback. If green is bad and blue is still healthy, moving the routing pointer back can be much faster than redeploying the old artefact from scratch.

That does not mean blue-green is safer by default. It means the shape of the risk changes. Rolling updates spread risk over time. Blue-green tends to concentrate the user-visible switch, but keeps the previous environment intact for a while.

That trade makes sense when you can answer yes to questions like these:

• can we afford two full application environments at once?
• can we verify green meaningfully before live traffic?
• can we keep shared state compatible long enough for rollback?
• can we stop side effects, workers, and schedulers from running twice?

If the answer to the shared-state question is no, then the deployment may still be worth doing, but the famous "instant rollback" promise has already weakened.

2. The switch point defines the whole deployment

People often say "we do blue-green" without specifying where the switch happens. That omission hides most of the engineering.

The switch can live at several layers:

A load balancer or reverse proxy cutover is the classic version because the operator controls it directly. If HAProxy, Envoy, nginx, or a cloud load balancer owns the traffic entry point, you can point the live backend set at green while keeping blue intact behind the same address.

A simplified HAProxy sketch looks like this:

backend checkout_live
    option httpchk GET /readyz
    server blue-1 10.20.0.11:8443 check
    server blue-2 10.20.0.12:8443 check
    server green-1 10.20.1.11:8443 check disabled
    server green-2 10.20.1.12:8443 check disabled

Before cutover, green is present but not eligible. During promotion, you enable green and drain blue rather than deleting blue outright.

In Kubernetes, the equivalent control often sits in a Service selector, an ingress route, or a higher-level progressive delivery controller:

apiVersion: v1
kind: Service
metadata:
  name: checkout
spec:
  selector:
    app: checkout
    track: green
  ports:
    - port: 443
      targetPort: 8443

That looks wonderfully simple. It is simple only if you remember what it actually changes. The Service does not teleport every in-flight stream onto a new pod. It changes which endpoints receive new connections or requests, subject to kube-proxy rules, ingress buffering, HTTP keep-alives, and whatever sits in front of the cluster.

DNS sounds attractive because it is easy to understand. Blue resolves to one IP, green to another, and you change the record. The trouble is that DNS is a hinting layer with caches you do not own. TTL helps, but TTL is not a guarantee that all clients and recursive resolvers will switch at the same second. Some stacks cache aggressively. Some operators put intermediate DNS appliances in front of their egress. Mobile apps can behave differently from browsers. CDN layers can add their own timing.

That is why DNS-based blue-green often behaves more like "eventual traffic migration" than like a single cutover event.

When someone says their rollback took twenty minutes even though the DNS TTL was thirty seconds, I do not assume the system is broken. I assume DNS was being asked to behave like a real-time traffic steering plane, which it is not.

3. Duplicating stateless compute is the easy part. Duplicating operational reality is harder

The cheerful diagram for blue-green usually shows two neat stacks labelled Blue and Green. Real systems do not come in neat stacks.

Application processes are only one part of production reality. A convincing green environment often needs all of these to line up with blue:

• the same secrets and certificate chains
• the same outbound network reachability
• the same message broker permissions
• the same object storage buckets or access policies
• the same observability pipeline and alert labels
• the same autoscaling rules or capacity reservations
• the same configuration sources and feature-flag baselines
• the same rate limits from external providers

Teams sometimes discover late that green was "healthy" only because its pre-production traffic volume never exercised a missing dependency. A classic example is a payment or SMS provider allow list. Blue's NAT egress IP is registered with the provider. Green's egress path is different. Synthetic probes pass because they only hit /readyz. The first real transaction after cutover fails at the provider boundary.

Another common mismatch is cache and storage locality. Green may be pointing at the same Redis cluster or the same object store, which is fine, but it may not have warm caches, compiled templates, JIT-optimised code paths, or connection pools yet. If the release is measured immediately after cutover, the team can mistake cold-start behaviour for a code regression.

This is why good blue-green preparation checks more than process liveness.

A green candidate should answer questions like these before promotion:

• can it terminate production TLS correctly?
• can it reach PostgreSQL, Redis, Kafka, and whatever else matters?
• can it call third-party APIs from the right source network?
• can it emit logs, metrics, and traces with the expected service identity?
• can it start enough workers or pods to absorb full traffic without autoscaling panic?
• can it run migrations or startup hooks without mutating shared state in a rollback-hostile way?

That last point matters because some systems do work at startup that looks harmless and is not. Examples include cache invalidation, queue lease acquisition, background index builds, one-off data backfills, and idempotency-key sweeps. If green performs those actions before it is meant to be live, you no longer have a quiet standby environment. You have a second actor in production clothing.

A blue-green plan is good when the green environment is boring. It should look like blue plus a new build, not like a laboratory full of one-off exceptions.

4. Green must be warmed before it is exposed

Health checks are necessary and weak.

A /readyz endpoint usually tells you that the process started, loaded config, and can answer a basic request path. That is useful. It does not tell you whether the environment is ready for a Frankfurt lunchtime burst, whether JVM or V8 code paths are warmed, whether connection pools have settled, whether the application has rebuilt its in-memory lookup tables, or whether the dependency graph behind the endpoint behaves the same under load.

The gap between "ready" and "ready for production traffic" is where many painful blue-green releases live.

Suppose a retailer in Milan deploys a new checkout service. Green starts twelve pods, every pod passes readiness in ten seconds, and the ingress considers them eligible. The team flips traffic immediately. Then latency spikes for four minutes because:

• the pods all build database connection pools at once
• the local caches are empty
• the JIT has not optimised hot code paths yet
• the autoscaler reacts after the burst rather than before it
• upstream rate-limited dependencies see a sudden fan-in from fresh connections

Nothing is technically broken. The promotion was still bad.

That is why mature blue-green workflows add warm-up stages before cutover:

1. boot green and let readiness go healthy
2. run synthetic probes through the real edge path
3. warm caches, templates, or lookup tables where possible
4. pre-establish expensive upstream connections
5. watch green-only metrics long enough to establish a baseline
6. only then make green eligible for user traffic

Some teams use shadow traffic for this, meaning a copy of production requests is replayed to green without the green response being returned to users. That can be valuable, but it must be understood clearly. Shadow traffic verifies parsing, code execution, and some dependency behaviour. It does not safely verify side effects unless the application and its dependencies are designed for shadow mode. A duplicated card authorisation or duplicate email send is not an acceptable probe.

The operational question is not "did the pod turn green in Kubernetes?" It is "if I send full traffic here in the next minute, what expensive surprises are still queued up?"

For systems with heavy caches, the warm-up phase may need its own controls. Examples include:

• running a read-only synthetic catalogue sweep to populate hot product keys
• precomputing compiled templates or rulesets
• seeding local machine-learning model weights into memory
• gradually opening upstream connection pools instead of letting every worker stampede at once

Blue-green works best when green has already finished its awkward first few minutes before the first real user sees it.

5. The cutover itself usually changes new traffic first, not old traffic

One of the most misleading phrases in release engineering is "switch all traffic". In many real systems, you do not switch all traffic. You switch the admission rule for new traffic, then wait while old traffic drains.

That distinction matters because protocols keep state.

If the cutover lives at an HTTP reverse proxy, new TCP connections can start going to green immediately. Existing keep-alive connections to blue may still carry requests until they idle out or the proxy deliberately closes them.

If the application uses HTTP/2 or gRPC, one client connection may multiplex many streams. An L4 or simple L7 cutover does not break that connection by magic. The stream owner often stays blue until the connection drains, the server sends GOAWAY, or the client reconnects.

If the product uses WebSockets, server-sent events, or long polling, the delay can be much longer. A dashboard browser tab open in Athens for three hours might remain attached to blue unless something forces reconnection.

This is why draining exists as a first-class deployment action.

A safe cutover often looks like this:

T0  green passes promotion gates
T1  router marks green eligible for new traffic
T2  router marks blue as draining, not dead
T3  new requests flow to green
T4  existing blue connections complete or are nudged closed
T5  blue reaches near-zero live sessions
T6  operator decides whether to retire blue or keep it warm for rollback

Draining is not an optional nicety. It is how you avoid turning a deployment into an outage for long-lived requests.

Proxy and gateway products expose this in different language:

• connection draining
• deregistration delay
• graceful shutdown timeout
• upstream GOAWAY
• termination grace period

The mechanism varies, but the intent is the same: stop giving blue new work, let existing work complete, and only then tear blue down.

There is a subtle tradeoff here. Long drain windows are kind to users, but they also keep blue meaningfully alive for longer. That can be good for rollback and bad for shared-state consistency if blue and green are not meant to process the same evolving data forever.

The right window depends on the protocol. Short JSON API calls might drain in tens of seconds. Report generation, file uploads, streaming endpoints, or WebSocket-heavy systems might need a more deliberate reconnect plan.

A system that cannot tolerate overlap between old and new application behaviour needs an answer to that protocol problem before claiming it does blue-green well. One more complication sits between routing and data storage: session locality.

If blue and green are truly interchangeable, a user should be able to reconnect to either colour and continue. Many systems are not built that way. They keep some state local to one application process or one node:

• in-memory session maps
• local template caches that affect latency sharply
• WebSocket subscription state
• temporary upload fragments on local disk
• per-process rate-limit buckets

Sticky sessions can hide this for a while. If the load balancer hashes a cookie and keeps a browser pinned to blue, the user may never notice that green cannot really resume the same interaction. The deployment still has a design flaw. Stickiness delays the moment the flaw becomes visible.

This matters most during rollback. Suppose green becomes unhealthy five minutes after cutover. The router starts sending new sessions back to blue. If user identity, basket state, or upload progress lived only in green memory, the rollback can restore HTTP availability while still breaking the user's task. From the operator's graph, the rollback succeeded. From the user's point of view, the checkout or document upload reset halfway through.

The safer patterns are familiar:

• keep user session state in shared storage rather than per-node memory
• treat WebSocket reconnection as a normal path, not an exceptional path
• store resumable upload state somewhere both colours can read
• avoid local-only caches that change correctness rather than just latency

This is also why blue-green deployment and feature flags solve different problems. A feature flag can keep one code path dark inside the same live fleet. Blue-green asks whether a whole environment can take over the live boundary. If the application still depends on local process memory for correctness, the cutover remains partially theatrical.

Statelessness is not a moral virtue here. It is an operational property that makes the colours genuinely swappable. The more correctness depends on which exact process handled the last request, the less clean the blue-green story becomes.

6. Databases decide whether rollback is real or just comforting language

For stateless compute, blue-green can feel elegant. For databases, it becomes adult engineering very quickly.

Most teams do not run two independent production databases and flip between them for every application release. Full dual-database blue-green can exist, but it brings replication, consistency, cutover, and data divergence problems of its own. In ordinary web systems, blue and green usually share the same database.

That shared database is where the famous rollback promise goes to be tested.

If green deploys against the shared database and performs a schema change that blue cannot tolerate, then traffic rollback may still be easy while application rollback is not. Blue is still standing there, but it no longer understands the truth in the database.

The safe pattern is the old expand-contract discipline:

1. add schema in a backward-compatible way
2. deploy code that can work with old and new shapes
3. cut traffic
4. verify stability
5. only later remove obsolete schema or old code paths

A harmless example:

ALTER TABLE orders
ADD COLUMN dispatch_deadline timestamptz NULL;

Old blue code can ignore the new nullable column. New green code can begin writing it. Rollback stays possible because the older binary still understands the row shape.

A dangerous example during the same release window:

ALTER TABLE orders
DROP COLUMN status_text;

If blue still reads status_text, rollback is no longer a routing command. It is a data-repair or re-deploy exercise.

The same problem appears with:

• enum changes older code cannot parse
• renamed columns or fields
• new validation rules that old code violates
• changed serialisation formats in a shared cache or event store
• irreversible backfills that overwrite old semantics

Event-driven systems make this sharper. If green publishes a new event shape to Kafka and old blue consumers cannot parse it, you may have created a rollback barrier without touching the SQL schema at all.

A practical deployment rule follows from this: do not spend your rollback budget on irreversible shared-state changes at the same moment you spend it on application cutover unless you truly mean to.

Many teams separate deployment into phases for this reason:

• first release: additive schema only
• second release: code that uses the additive schema
• later cleanup: remove old schema once the rollback window is closed

That feels slower than a one-shot migration. In exchange, rollback remains real.

The database is often the reason senior operators sound sceptical when someone says, "blue-green means instant rollback". Their hidden follow-up is, "for which layer?"

7. Background workers, schedulers, and queues can make both colours live in the dangerous sense

HTTP traffic is only one way production work happens.

Many systems also have:

• queue consumers
• scheduled jobs
• compaction or reconciliation workers
• email or notification senders
• payment retry loops
• ledger settlement tasks
• cache refresh daemons

If green comes up with those workers enabled before cutover, blue-green is no longer a neat one-live-one-standby story. Both colours are now producing side effects.

This is one of the nastiest release mistakes because it can stay invisible until the wrong workload fires.

Imagine a queue-backed invoicing worker. Blue is consuming from invoice.generate. Green is deployed and the operator forgets to gate the workers. Now both colours consume from the same queue. If the jobs are not perfectly idempotent, duplicate PDFs, duplicate emails, or duplicate provider calls appear. Even if the jobs are idempotent, throughput and ordering can shift in unexpected ways.

Or imagine a scheduled cleanup job. Blue and green both wake at midnight Central European Summer Time and decide they own the retention sweep.

The fix is conceptually simple: make side-effecting workers an explicit part of the colour state machine.

Typical controls include:

• enable queue consumers only for the live colour
• move scheduler leadership behind a lease or lock
• separate read-only warm-up jobs from mutating jobs
• label emitted events with deployment colour and build identity
• make external side effects idempotent where the domain allows it

A lease-based pattern can be as simple as one active owner record:

worker_role = settlement-runner
owner      = blue
lease_ttl  = 30s

During cutover, the owner moves to green only after green is verified. During rollback, the lease can move back. The point is not the exact storage mechanism. PostgreSQL advisory locks, Redis leases, ZooKeeper-style coordination, or cloud-native leader election can all do the job. The point is that one colour must have an explicit right to perform the side effect.

This is where application architecture meets deployment style. If the system was built with idempotency, leases, and explicit job ownership, blue-green is operationally plausible. If jobs are implicit, side effects are fire-and-forget, and cron runs everywhere by default, blue-green becomes much more dangerous than the marketing diagram suggests.

8. Rollback is a shrinking window, not a permanent property

The nicest blue-green story is this: green goes live, error rate jumps, the operator flips traffic back to blue, and the incident ends in under a minute.

That story can be true. It is true only while a set of hidden conditions still hold.

Rollback stays easy when:

• blue is still running and warm enough to take traffic
• the router change is under your control
• blue and green still understand the same shared data
• green has not emitted irreversible side effects that need repair
• session and connection handling allow users to reconnect cleanly

Those conditions decay over time.

Blue may be scaled down to save cost.

New schema or event formats may make blue obsolete.

A third-party side effect may already have escaped. If green already captured payments, sent emails, or altered inventory state, traffic rollback cannot erase those facts.

Cache invalidation or derived data may already have moved the system into a new steady state that blue can serve only partially.

This is why good operators talk about a rollback window rather than about rollback as a timeless capability.

The window can be short. For some systems it is measured in minutes. Once a destructive migration runs, or once blue is torn down, or once queue ownership fully transfers and the new version writes a shape the old version cannot parse, the rollback window closes.

DNS makes this more awkward because the router is not purely yours. If you changed a record from blue to green and then change it back, the old record can reappear gradually as caches age out. Users can remain split between colours during the rollback itself.

Load-balancer-based cutover is usually more decisive. If you own the gateway, you can often reject new green admissions immediately and send fresh traffic back to blue while green drains. Even then, the shared-state caveat still rules.

A useful operational question during release planning is:

At what exact moment does this rollout stop being a one-command rollback and start being an incident response procedure?

If the answer is vague, the rollback plan is vague.

9. Observability for blue-green is about differences between colours, not just overall health

A release can look healthy in aggregate while one colour is clearly worse.

Suppose total p95 latency remains acceptable because only 15 percent of traffic is still pinned to blue during drain, or because green is fast for catalogue reads and terrible for checkout writes that represent a smaller slice of volume. Aggregate dashboards can blur that difference.

Blue-green observability needs colour-aware comparisons.

At minimum, during the release window you want to break down metrics by:

• deployment colour
• build or image revision
• pod or instance group
• upstream dependency where relevant
• request class or route family

Useful questions include:

• is green's error rate higher than blue's for the same route?
• are green's cold cache misses falling as expected, or staying high?
• did connection count move cleanly from blue to green after cutover?
• are background jobs still running only in the intended colour?
• are synthetic probes through the public path still succeeding after the real cutover?

Tracing helps here because it shows whether the new version changed the structure of work, not just its duration. A green trace might reveal an extra database call, a missing cache hit, a different retry loop, or a shifted outbound dependency path.

Logs matter too, but release-time logging must be queryable by build and colour. "grep the fleet" is not a deployment control plane.

One practical pattern is to gate each release phase on a short, explicit metric checklist. For example:

Promote green only if, over the last 5 minutes:
- synthetic checkout success >= 99.9%
- green p95 latency within 10% of blue baseline
- green 5xx rate <= blue 5xx rate + 0.05%
- queue lag unchanged for active worker roles
- no new high-severity logs by build revision

The point is not the exact thresholds. The point is that blue-green should be instrumented as a controlled experiment with one environment replacing another, not as a vague hope that the global graph stays roughly flat.

10. Blue-green is expensive because it buys certainty at cutover time

The obvious cost of blue-green is duplicate capacity. For some period, you run two application environments where one would normally do.

That cost is real, but it is only the first line item.

The fuller cost picture includes:

• spare compute for the standby environment
• warm caches and pre-opened connections in green
• storage or queue capacity if green replays or shadows traffic
• observability work to compare colours properly
• engineering discipline around schema compatibility and job ownership
• operational time spent rehearsing cutover and rollback

In other words, blue-green is not expensive because somebody likes neat diagrams. It is expensive because it moves uncertainty out of the user-visible switch and into preparation, compatibility, and spare capacity.

That can be a very good trade.

If your system has a small number of high-value releases, strong rollback requirements, and a user-facing entry point that you control cleanly, blue-green can make incidents shorter and diagnosis easier.

If your system releases dozens of times per day, uses additive compatible changes almost exclusively, and already has good rolling or canary controls, the extra duplicate-capacity model may not be worth it for every service.

This is where deployment styles should be chosen mechanically rather than ideologically.

Blue-green is strong when you want:

• a crisp old-versus-new comparison
• a clean application-tier fallback path
• minimal mixed-version serving time
• a controlled release ceremony for a high-risk change

Canaries are stronger when you want:

• gradual exposure by user percentage
• real production feedback before full blast radius
• more time to detect subtle behavioural regressions

Rolling updates are efficient when you want:

• low spare-capacity overhead
• simple automation
• frequent small releases with limited operational ceremony

Many mature organisations use all three, depending on service shape and risk class. Blue-green is not the final form of deployment maturity. It is one well-defined tool.

11. One concrete cutover trace makes the mechanics easier to see

Let us walk a concrete release path for an imaginary European retailer.

The service is checkout-api. It runs in Kubernetes in Dublin. Public traffic enters through an L7 gateway backed by Envoy. Blue currently serves production. Green is the new release candidate.

Shared dependencies:

• PostgreSQL for orders
• Redis for idempotency keys and cart cache
• Kafka for downstream event publication
• Adyen for payment authorisation
• an email service for receipts

Release plan:

1. Apply additive schema migration:

ALTER TABLE orders
ADD COLUMN fraud_context jsonb NULL;

Blue ignores the new column. Green can start writing it.

2. Deploy green pods with workers disabled. They boot, pass readiness, establish database pools, and warm hot route caches from synthetic read-only probes.

3. Run synthetic transactions end to end through the real gateway. Payment provider calls are exercised in a safe sandbox path, not on live cards.

4. Compare green telemetry to blue telemetry for ten minutes. No production user traffic yet.

5. Mark green upstreams active in Envoy. Mark blue as draining.

At this point, what actually happens?

• new TLS sessions and new HTTP admissions start landing on green
• some older browser connections remain on blue until idle timeout or GOAWAY
• the scheduler lease for email-receipt workers is still blue until checkout success and queue lag remain normal for a few minutes
• PostgreSQL sees writes from both colours temporarily, but both binaries understand the schema

Seven minutes later, operators see green p95 latency spike on checkout confirmation. Trace comparison shows an accidental extra Kafka publish on the success path. Error rate is still low, but queue lag and request time are rising.

Because the migration was additive and blue is still warm, the rollback is still real.

The operator does three things:

1. stop new traffic admissions to green at the gateway
2. keep blue active for new traffic
3. keep green up long enough to inspect logs and drain in-flight requests

Blue resumes serving new traffic almost immediately. Green remains available for diagnosis but no longer owns the live path. No database repair is needed because the new fraud_context column is additive and blue tolerates it.

Now change one detail. Suppose the same release had renamed an event field that old consumers could not parse, or dropped a column blue still required. Traffic rollback might still be possible at the gateway, but application rollback would already be compromised. That is the difference between a real blue-green rollback and a comforting story about one.

12. Blue-green deployment is routing discipline backed by compatibility discipline

The router command is the visible part. The compatibility work is the part that makes the visible command safe.

A blue-green deployment succeeds because:

• green is a real production candidate, not a smaller imitation
• the cutover point is understood precisely
• warm-up happens before exposure, not during the first user burst
• old connections drain deliberately
• database and event contracts stay rollback-compatible long enough
• workers, schedulers, and side effects have a single active owner
• colour-aware observability tells you what changed, not just whether the fleet still responds

When those pieces are in place, blue-green can be excellent. The live switch is crisp, diagnosis is easier, and rollback for the application tier can be very fast.

When those pieces are missing, blue-green becomes cargo-cult release engineering. Two colours exist in the diagram, but the real system still has mixed versions, incompatible shared state, duplicated side effects, and a rollback plan that died the moment the migration ran.

That is the practical summary.

Blue-green deployment is not magic. It is controlled traffic admission plus disciplined state compatibility. The routing flip is easy. Earning the right to flip it is the real work.