How Reverse Proxies Actually Work

Most engineers first meet a reverse proxy through a tiny config snippet:

location / {
    proxy_pass http://app;
}

That makes the subject look smaller than it is. A reverse proxy is not just a pipe with a hostname. It is the public protocol endpoint that stands in front of one or more origin services and decides what the application will actually see. The client opens a connection to the proxy, not to the app. TLS may terminate there. HTTP may be parsed, normalised, rewritten, buffered, retried, cached, rate-limited, logged, or rejected there. Upstream connections may have different lifetimes, different protocol versions, and different trust assumptions from the client-facing side.

That is why reverse proxies sit in so many important places:

• nginx or HAProxy in front of a monolith
• Envoy in a Kubernetes ingress
• a CDN edge proxy in front of an origin
• an API gateway in front of internal services
• a service-mesh sidecar in front of one process

The details change. The mechanism stays the same. A reverse proxy accepts a request on behalf of a server-side service, makes a new request or stream decision toward an upstream, and enforces policy at that boundary.

If that boundary is configured well, the application gets a calmer, clearer world. It sees fewer client TCP handshakes, fewer slowloris-style stalls, fewer pointless reconnects, fewer duplicate upstream fetches, and more consistent metadata. If that boundary is configured badly, the application logs the wrong client IP, generates http:// redirects behind HTTPS, retries unsafe writes, buffers uploads to disk until the node fills, breaks WebSocket upgrades, or becomes vulnerable to parser mismatch attacks.

This article walks through the machinery that matters: where the first connection terminates, how routing rules are evaluated, how forwarded identity is propagated, why connection pooling and buffering change backend behaviour, where retries and timeouts belong, what kinds of transformations proxies perform, and which failure modes show up in production most often.

A reverse proxy is the server-side face of a service

A forward proxy represents the client. A reverse proxy represents the service.

That sounds simple. It matters because it tells you where the trust boundary lives.

A short comparison helps:

A reverse proxy often performs load balancing, and a CDN is a specialised reverse-proxy fleet, but the concepts are not identical.

The key point is this: the public DNS name usually resolves to the reverse proxy layer, not to the application process that ultimately produces the response body. When a browser visits https://app.example.eu, it is really talking to an edge component that may later choose among several upstreams:

browser
  -> reverse proxy
      -> app-a
      -> app-b
      -> static cache
      -> auth service

That proxy layer exists because direct application exposure creates a rough operating environment:

• every app instance must own public TLS and certificate rotation
• every app instance must absorb raw internet connection churn
• every app instance must defend itself against slow readers and slow writers
• routing by host, path, or method becomes application code instead of infrastructure policy
• connection reuse and backpressure policy become fragmented across runtimes

Putting a reverse proxy in front does not magically solve these problems. It gives you one deliberate place to solve them.

This is also why a reverse proxy is rarely "transparent" in the strict sense. Even if the payload body is untouched, the protocol context is not. Headers change. Connection lifetime changes. TLS state changes. Source address visibility changes. Timeouts change. The proxy is part of the system's semantics.

The first connection ends at the proxy, not at the application

The most important mental model is that a reverse proxy does not forward one connection unchanged through to the backend. It terminates one side and originates another.

For a normal HTTPS request, the path often looks like this:

Client in Athens
  -> TCP handshake with proxy VIP
  -> TLS handshake with proxy certificate
  -> HTTP request over that TLS session
  -> proxy parses request and selects upstream
  -> proxy opens or reuses upstream connection
  -> proxy sends upstream request
  -> upstream response returns to proxy
  -> proxy streams or buffers response back to client

There are two separate protocol relationships here:

1. client to proxy
2. proxy to upstream

They can differ substantially.

A browser may speak HTTP/2 over TLS 1.3 to the public proxy. The proxy may then speak HTTP/1.1 with keep-alive to the origin service. Or it may speak HTTP/2 upstream. Or it may speak FastCGI, uwsgi, gRPC over HTTP/2, or even raw TCP in a stream proxy mode.

That translation boundary is powerful because it lets the proxy absorb protocol complexity or internet-facing risk while keeping the backend simpler. A classic example is front-door HTTP/2 with upstream HTTP/1.1. Browsers get multiplexing and header compression on the public side. The application keeps a familiar request model behind the proxy.

TLS termination is a real endpoint, not a cosmetic step

When the proxy terminates TLS, it owns several jobs the backend never sees directly:

• SNI selection and certificate presentation
• ALPN negotiation such as HTTP/1.1 versus HTTP/2
• cipher and protocol version policy
• client-certificate validation if mTLS is used at the edge
• handshake timeout and abuse controls

That means a request can fail before the application knows it existed.

Examples:

• the client offers only obsolete TLS versions
• the requested hostname has no matching certificate
• the client sends a huge header block and hits proxy limits
• the client stalls mid-request and trips a read timeout

From the app's perspective, none of those are application errors. They are edge admission decisions.

One public connection may map to many upstream requests

Under HTTP/2 or HTTP/3, one browser connection can carry many concurrent requests. The reverse proxy terminates that session, tracks streams, then decides how those requests map onto upstream resources.

That changes backend economics.

Without a proxy, 2,000 active browser connections might mean 2,000 directly exposed app connections. With a proxy, those same users might produce a far smaller and steadier pool of upstream keep-alive connections. The proxy becomes a shock absorber between internet fan-out and application concurrency.

The proxy must understand hop-by-hop semantics

HTTP has headers and connection semantics that are specific to one hop and must not be blindly relayed as if they were end-to-end metadata.

In HTTP/1.1, the classic hop-by-hop set includes:

• Connection
• Keep-Alive
• Proxy-Authenticate
• Proxy-Authorization
• TE
• Trailer
• Transfer-Encoding
• Upgrade

A reverse proxy has to consume or reinterpret these at the boundary. If it just copies bytes naïvely between parser domains, it can create request framing ambiguity or broken upgrade behaviour.

This is one reason proxy software is harder than it looks. It is not doing string replacement on HTTP. It is maintaining two correct protocol conversations at once.

Routing starts after the proxy has a parsed request, not before

Once the proxy has accepted the client-side request, it can make decisions using application-layer information.

At that point it knows things an L4 balancer does not:

• hostname
• path
• method
• query string
• selected headers
• cookies
• protocol version
• sometimes TLS properties such as SNI or client certificate identity

That makes much richer routing possible.

A common pattern is simple host and path splitting:

upstream web_pool {
    server 10.20.0.11:8080;
    server 10.20.0.12:8080;
    keepalive 128;
}
 
upstream api_pool {
    server 10.20.1.21:9000;
    server 10.20.1.22:9000;
    keepalive 256;
}
 
server {
    listen 443 ssl http2;
    server_name app.example.eu api.example.eu;
 
    location /assets/ {
        root /srv/public;
        expires 1h;
    }
 
    location /v1/ {
        proxy_pass http://api_pool;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Request-Id $request_id;
    }
 
    location / {
        proxy_pass http://web_pool;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Request-Id $request_id;
    }
}

The request to /assets/logo.svg may never touch the app. The request to /v1/orders/42 may land in the API pool. The request to / may land in the web pool. One public server name can fan into several backend topologies.

Rewrites are part of the routing contract

Reverse proxies often rewrite the request they send upstream:

• strip a prefix such as /api/
• change the Host header
• add a tenant or environment header
• normalise or reject suspicious paths
• attach a request ID for tracing

Suppose the public route is:

GET https://shop.example.eu/api/v1/orders/42

The backend might actually expect:

GET /v1/orders/42
Host: orders.internal.svc

The proxy is therefore not just choosing an upstream. It is adapting one externally stable interface to an internal one.

That adaptation needs discipline. Path rewriting bugs cause entire categories of production failure:

• upstream generates redirects to internal paths
• signature verification fails because the proxy changed the path after the client signed it
• application ACLs assume a prefix is present when the proxy already removed it
• cache keys differ between layers because the public and internal paths are not the same resource model

WebSocket and upgrade traffic expose the boundary clearly

Ordinary HTTP request routing is stateless compared with upgraded or long-lived protocols.

For a WebSocket request, the client starts with an HTTP handshake containing:

Connection: Upgrade
Upgrade: websocket

The proxy must recognise this as a state transition, preserve the necessary headers, and switch into a bidirectional streaming mode once the 101 Switching Protocols response arrives. A proxy configuration that works for JSON APIs can still break WebSockets if buffering or upgrade handling is wrong.

That is an important pattern. Reverse proxies are not merely route selectors. They are protocol-state machines.

Forwarded identity is useful, and it is easy to get wrong

Applications very often need to know something about the original client even though the immediate TCP peer is the reverse proxy.

The usual questions are:

• what IP did the user really come from?
• was the public request HTTPS?
• what hostname did the user actually request?
• which proxy chain handled the request before it reached the app?

The standard and de facto answers are forwarded headers.

A common backend request after proxying looks like this:

GET /v1/orders/42 HTTP/1.1
Host: api.example.eu
X-Forwarded-For: 203.0.113.44, 10.20.0.15
X-Forwarded-Proto: https
X-Forwarded-Host: api.example.eu
X-Request-Id: c3e4d2ac5f9a41b7

The first IP in X-Forwarded-For is typically the original client. Each trusted proxy that appends to the chain adds its own address after it. RFC 7239 defines the more structured Forwarded header, but many stacks still rely primarily on the X-Forwarded-* family for compatibility.

Trust boundaries matter more than header names

The dangerous mistake is assuming that because a header is named X-Forwarded-For, it must be truthful.

It is only trustworthy if all of the following are true:

• the application receives traffic only from known proxy peers
• untrusted clients cannot connect directly to the backend
• the proxy overwrites or appends the header in a controlled way
• the framework is configured with the correct trusted-hop policy

Otherwise the client can inject fake identity.

A backend that blindly trusts X-Forwarded-Proto: https may believe the original request was secure even if a direct attacker connection supplied that header manually. A rate limiter that trusts a spoofed X-Forwarded-For value becomes easy to evade. An audit log that records attacker-supplied IP data becomes forensic noise.

This is one of the most common reverse-proxy design bugs in application stacks. The application code looks fine in isolation. The deployment path makes it false.

Host and scheme reconstruction break more software than people expect

Applications routinely need to generate absolute URLs, redirects, callback endpoints, cookie policies, and origin checks. If the proxy does not pass correct public-origin context, those features break in subtle ways.

Typical symptoms:

• backend redirects to http:// behind an HTTPS site
• cookies miss the Secure attribute because the app thinks the scheme is plain HTTP
• OAuth callback URLs point at an internal hostname
• CSRF or origin checks compare against internal rather than public hostnames

That is why X-Forwarded-Proto, X-Forwarded-Host, and trusted proxy settings in frameworks matter operationally, not cosmetically.

Sometimes the right tool is PROXY protocol, not HTTP headers

HTTP headers only help once the upstream protocol is HTTP-aware. If the next hop is a TCP service, a TLS terminator, or another proxy that needs the original connection metadata before any HTTP parsing, PROXY protocol is often used instead.

HAProxy's PROXY protocol prepends a small metadata frame to the downstream connection so the receiving service learns the original source and destination addresses. Version 1 is text based. Version 2 is binary and supports extra metadata.

That is common in chains such as:

cloud load balancer
  -> HAProxy or nginx stream listener with PROXY protocol
      -> TLS terminator or app-aware proxy

Again, the principle is the same. Once a proxy stands between client and server, original identity must be reintroduced intentionally.

A reverse proxy shields the backend from internet connection behaviour

One of the least appreciated jobs of a reverse proxy is smoothing the mismatch between messy client behaviour and the backend's preferred operating conditions.

Internet clients are noisy:

• they connect and disconnect unpredictably
• they read at different speeds
• they send headers slowly
• they abandon half-finished uploads
• mobile networks flap between good and bad paths
• browsers can multiplex many logical requests onto a few connections

Application processes usually prefer:

• a bounded number of warm upstream connections
• complete request bodies, or clearly failed ones
• stable request framing
• predictable timeouts
• less exposure to slow readers and writers

The proxy helps by terminating the unstable side and maintaining a calmer upstream side.

Connection pooling reduces backend churn

Suppose 10,000 clients each make small HTTPS API calls over a minute. Without a proxy, the app fleet may face a flood of handshakes, kernel state transitions, and short-lived sockets. With a reverse proxy, many of those clients terminate at the edge, while the proxy reuses a smaller pool of upstream keep-alive connections.

A simplified view:

10,000 client-side TCP or TLS sessions
  -> 300 warm proxy-to-app keep-alive sessions

The exact numbers vary. The operational effect is real:

• fewer backend accept calls
• less TLS work if TLS terminates at the edge
• lower connection-establishment latency to the app
• better reuse of congestion window and socket buffers on warm upstream links

This matters even more for runtimes that are good at request handling but not ideal as raw internet socket endpoints.

Buffering protects backends from slow clients

Request buffering means the proxy can read part or all of the client request body before pushing it upstream. Response buffering means it can read from the backend faster than the client consumes bytes and then drip the response out to the slow reader.

That is useful because it decouples two very different clocks.

Example:

• a client on a poor train WiFi link uploads a 200 MB video slowly
• the backend would rather not keep an application worker tied to that slow upload socket
• the proxy can absorb the slow client-side pacing and deliver a steadier stream upstream, or reject the upload based on size before it reaches the app

On the response side:

• the backend can send a 2 MB JSON export quickly to the proxy
• the proxy can hold the bytes and feed them to a slow client at the client's pace
• the backend worker becomes free sooner

This shielding is one reason nginx became so widely used in front of app servers with expensive per-request concurrency models.

Buffering is not always the right answer

The tradeoff is that buffering consumes memory and, for larger bodies, often disk.

If request buffering is enabled and the body exceeds in-memory thresholds, the proxy may spill to temporary files. That is reasonable for controlled uploads. It becomes dangerous if operators forget the storage implications.

A few common surprises:

• one node runs out of ephemeral disk because large uploads are buffered locally
• a streaming API becomes non-streaming because the proxy waits for the full body
• a gRPC or server-sent-events path stalls because response buffering was left on

That is why serious configurations split behaviour by route. The upload endpoint may need different buffering policy from ordinary JSON requests.

HTTP version translation changes backend pressure

A browser speaking HTTP/2 can send many streams concurrently over one connection. The proxy can then spread those streams across several upstream HTTP/1.1 keep-alive connections, or multiplex them again over upstream HTTP/2 if supported.

That means the backend does not necessarily experience client concurrency in the same shape the client generated it.

This is helpful, but it can also hide bottlenecks. The public side may look healthy while the upstream connection pool quietly saturates. When that happens, requests queue at the proxy and the problem shows up as added upstream connect or header wait time rather than obvious TCP failure.

Timeouts, retries, and draining are where reverse proxies become reliability policy

Once a reverse proxy owns both ends of the request path, it is in a position to decide what kinds of failure should be retried, how long each phase may take, and when a backend should stop receiving new work.

This is where the software stops being a router and becomes operational policy.

One request has several distinct timeout budgets

A decent proxy does not have just one timeout called "request timeout". It usually tracks several phases:

• client header read timeout
• client body read timeout
• upstream connect timeout
• upstream response header timeout
• upstream idle read timeout
• send timeout toward the client

These phases fail for different reasons.

• A short upstream connect timeout catches dead routes or overloaded accept queues.
• A header timeout catches an app that accepted the connection but is not producing a response.
• A client read timeout catches slowloris behaviour or broken upload senders.
• A send timeout catches clients that stopped reading.

Lumping them together makes debugging much harder.

Retries are powerful because they can also be dangerous

If the chosen upstream fails before a response is committed, the proxy may retry another backend. This is often correct for idempotent requests such as GET, HEAD, or some carefully designed PUT operations.

It is often wrong for unsafe writes.

Suppose an order-creation API receives:

POST /v1/orders

The first backend writes the order to PostgreSQL, then crashes before sending the response. From the proxy's point of view, that may look like an upstream failure. If the proxy blindly retries the POST to another backend, the system may create the order twice unless the application has its own idempotency key logic.

That is why retry policy must account for:

• HTTP method
• whether any upstream bytes were already sent
• whether request bodies are replayable
• whether the application operation is idempotent by design
• whether a response has already started flowing to the client

A safe-looking retry is sometimes a distributed write duplication bug in disguise.

Draining lets old backends leave without cutting live work

During deployments, certificate rotation, or node maintenance, operators often want a backend to stop receiving new requests while allowing old ones to finish.

That is draining.

For short HTTP requests the effect is small. For long uploads, Server-Sent Events, WebSockets, and gRPC streams, it is essential.

A reverse proxy that supports draining properly can:

• remove a backend from new selection
• keep existing long-lived streams attached for a grace period
• close idle keep-alives first
• expose metrics that show when it is truly safe to terminate the instance

Without draining, a deployment becomes a coin toss between speed and user-visible resets.

Health is not binary

Most reverse proxies support active health checks, passive failure observation, or both.

The blunt model is "up" or "down". Real incidents are rarely that clean.

A backend may:

• answer /healthz with 200 OK
• accept TCP quickly
• still have a saturated database pool that makes real requests take 4 seconds

From the user's point of view, that backend is harmful even though it is technically alive.

This is why modern proxy layers often include richer behaviour such as:

• passive ejection after repeated 5xx responses
• latency-aware outlier detection
• separate health endpoints for readiness versus liveness
• circuit-breaker style concurrency caps

The reverse proxy is not the whole reliability system, but it is one of the earliest places that can notice backend quality and reduce blast radius.

Reverse proxies often transform responses, cache content, and enforce policy

Many teams think of reverse proxying as request forwarding plus load balancing. In practice, the layer often does much more because it already owns both protocol ends.

Static responses and cache hits may never reach the app

A reverse proxy can answer directly when the response is:

• a static asset on local storage
• a cached upstream response
• a synthetic redirect or error page
• a health endpoint generated by the proxy itself

That means the application may not even know a user asked for a given object.

Caching is the most economically important case. For a cacheable API or page response, the proxy can store the upstream result and serve later requests without waking the backend.

A simple origin response might be:

Cache-Control: public, s-maxage=60, stale-while-revalidate=15
ETag: "orders-9f21"

A reverse proxy that honours shared-cache semantics can then:

• serve later requests from cache for 60 seconds
• continue serving a slightly stale copy for 15 more seconds while revalidating
• protect the origin from duplicated identical fetches

This is one reason reverse proxies and CDNs overlap conceptually. A CDN is a globally distributed version of the same idea.

Compression and normalisation live naturally at the edge

Because the proxy sees request headers and response bodies, it is often a good place to perform:

• gzip or brotli compression
• header normalisation
• canonical redirects such as www to apex or HTTP to HTTPS
• security-header insertion
• response-size limits
• simple bot or abuse filtering

That can simplify the app, but it can also create confusion about source of truth.

If the proxy injects Strict-Transport-Security, CSP, or X-Content-Type-Options, operators must remember that the behaviour is now edge-defined. App developers reading only application code may not know why browsers see certain headers.

Request and response transformation can break signatures and caches

Transformation is useful. It is also where subtle bugs live.

Examples:

• a proxy decompresses a body but the application expected to verify a signature over the compressed bytes
• a proxy lowercases or rewrites headers that a downstream auth scheme treats as canonical input
• a proxy strips a path prefix after an upstream cache key was designed around the public path
• a proxy injects a cookie and accidentally destroys cacheability for shared responses

This is the recurring theme of reverse proxies: any feature that makes the boundary useful also makes the boundary semantically important.

Queueing and overload handling decide whether the proxy helps or amplifies trouble

Every reverse proxy is also a resource manager. It has a finite number of workers, sockets, buffers, queue slots, header parsers, temporary files, and upstream connections. When traffic rises or a backend slows down, the proxy has to decide what happens to excess work.

That decision is rarely visible in the happy path. It becomes decisive during overload.

Imagine an API pool that normally answers in 25 ms and can comfortably sustain 200 concurrent upstream requests per node. Then one shared dependency slows down and each request starts taking 900 ms instead.

The arithmetic changes immediately.

• the proxy's upstream connections stay occupied much longer
• new client requests keep arriving at the old rate
• request queues begin to form in front of the upstream pool
• client-side timeouts and retries start stacking on top of the original slowdown

If the proxy accepts unbounded waiting, it turns one slow dependency into a backlog amplifier. Users do not just see the original 900 ms problem. They see queue delay plus backend delay, often followed by a timeout anyway.

This is why good reverse-proxy behaviour under stress is often deliberately unfriendly. It may reject work quickly instead of pretending it can hold everything.

Typical controls include:

• per-upstream connection caps
• pending-request queue limits
• header and body size limits
• per-route concurrency limits
• fast failure with 503 Service Unavailable when the queue is already unhealthy
• separate policies for cheap reads versus expensive uploads or streaming routes

The principle is simple: a bounded queue can protect the rest of the system. An unbounded queue usually converts latency trouble into memory trouble and then into wider failure.

Backpressure is part of the proxy contract

Backpressure means telling the sender, directly or indirectly, that the receiver cannot keep absorbing work at the current rate.

At a reverse proxy boundary, that can show up as:

• delaying reads from the client because the upstream side is full
• refusing new upstream requests once pool limits are reached
• returning 429 or 503 rather than queueing indefinitely
• shedding low-priority routes before high-priority ones

This matters because the proxy is often the first place that can see both sides of the pressure gradient:

• how quickly clients are sending
• how quickly upstreams are clearing work

An application instance may only know that its own queue is growing. The proxy can see the broader pattern across the whole pool and act earlier.

Overload policy should differ by route

Not all requests deserve the same treatment.

A proxy can often reject or defer a low-value analytics call far more aggressively than a payment authorisation request. A 20 MB upload should not compete for the same buffering budget as a tiny cacheable GET. A long-lived event stream should not share idle timeout policy with a one-shot JSON fetch.

That is why mature configurations split policy by path or upstream cluster rather than using one global default for everything.

Examples:

• /metrics may have tiny timeouts and no retries
• /uploads/* may have large body limits but tighter concurrency caps
• /api/payments/* may allow almost no automatic retries without idempotency keys
• /events/* may disable response buffering and raise idle timeouts

The route matcher is already present for proxying. Using that same boundary for overload policy is one of the reasons the reverse proxy is such a useful control point.

A proxy can become the bottleneck if you ignore its own limits

Teams sometimes focus so much on protecting the backend that they forget the proxy is software with its own ceilings.

Under pressure, the proxy may run out of:

• worker CPU for TLS and parsing
• memory for outstanding buffered bodies
• file descriptors for connections
• local disk for temporary files
• fairness if one route monopolises upstream slots

This is why proxy dashboards often need at least these signals:

• active downstream connections
• active upstream connections
• pending requests or queue depth
• upstream response time percentiles
• request rejection counts by reason
• temporary-file or buffer spill metrics

If those counters are missing, operators tend to blame "the app" for symptoms that actually began in the admission layer above it.

A reverse proxy does not eliminate overload. It decides how overload is shaped. That is a major difference. When the policy is sane, the system fails in a smaller and more diagnosable way. When the policy is vague, the proxy becomes a latency warehouse that stores doomed work until everything is slower at once.

Modern deployments often stack several reverse proxies on one request path

The old picture was one nginx box in front of one application. That still exists. Modern paths are usually layered.

A single user request may traverse several reverse-proxy roles:

browser
  -> CDN edge proxy
  -> regional shield or origin proxy
  -> Kubernetes ingress proxy
  -> sidecar proxy
  -> application container

Each hop may add or interpret different metadata.

• the CDN may terminate public TLS and add a client IP header
• the ingress may route by host and path inside the cluster
• the sidecar may apply mTLS and retries toward the service process

This layering is useful because each component solves a different problem at a different boundary. It is also why debugging proxy-heavy stacks can become tedious. When a header is wrong or a timeout fires, you have to know which hop owned that behaviour.

API gateways and ingress controllers are specialised reverse proxies

An API gateway is usually a reverse proxy with a wider control-plane and policy surface:

• authentication and authorisation hooks
• developer-facing routing products
• quota enforcement
• request transformation and version mediation
• tenant-aware traffic policy

A Kubernetes ingress controller is also fundamentally a reverse proxy. It watches cluster objects and generates routing configuration from them.

Envoy, nginx, HAProxy, Traefik, and cloud-managed gateways all sit somewhere on this spectrum. The implementation details differ. The underlying mechanism remains reverse proxying.

Service meshes move the proxy boundary closer to each process

With a sidecar mesh, the reverse proxy may not be a public edge component at all. It may sit next to each service instance and own east-west traffic policy inside the cluster.

That changes the scale and the concerns:

• public TLS termination may happen elsewhere
• retries may now occur between internal services
• mTLS identity may be per workload rather than per public hostname
• observability and traffic shaping become much more granular

The same core mechanics still apply: terminate one hop, start another, preserve identity intentionally, and own timeout plus retry semantics explicitly.

The ugliest production failures happen at proxy boundaries

Reverse proxies are powerful. Their failures are unusually good at wasting days because the symptoms often show up in the application while the cause lives in the proxy layer.

Here are some of the failures that recur in real systems.

Wrong client IP and broken security decisions

If the app trusts the wrong forwarded header policy, you get:

• fake audit source IPs
• useless geo checks
• broken abuse controls
• incorrect rate limiting
• misleading fraud signals

The fix is architectural, not cosmetic. Ensure backends are reachable only from trusted proxies, overwrite identity headers at the edge, and configure trusted-hop counts precisely.

Redirect loops and mixed-scheme confusion

If the proxy terminates HTTPS but the app thinks it is on HTTP, the app may redirect every request back to HTTPS even though the client is already there. Some stacks produce an infinite 301 or 308 loop. Others emit absolute URLs with the wrong scheme and break cookies or callback flows.

This nearly always comes down to public-origin reconstruction: Host, scheme, trusted proxy settings, and application URL generation rules must agree.

Large request bodies fill proxy disk

A file-upload route may work perfectly in staging and then fail under real usage because request buffering spills to local temporary storage. Suddenly one node's root volume is 100% full and unrelated requests start failing.

You do not solve that by shouting at the uploader. You solve it by deciding intentionally:

• should this route be buffered at all?
• where do temp files live?
• what size limits are acceptable?
• do we stream directly to object storage instead?

Long-lived upgrades die on idle defaults

WebSockets, SSE, and gRPC streams are very good at exposing hidden timeout defaults.

A proxy configured for ordinary REST traffic may have:

• idle timeouts that are too short for a chat connection
• buffering rules that break streaming semantics
• header limits that do not match gRPC metadata reality
• draining windows too short for long-lived sessions

The application team often sees "random disconnects". The proxy team sees a completely reasonable default for short requests. Both are technically correct until they align on the real traffic shape.

Parser mismatch can become a security bug

When two HTTP parsers disagree about framing, header folding, duplicated length fields, or transfer-encoding handling, a malicious request can be interpreted differently by the front proxy and the backend. That is the family of issues behind HTTP request smuggling.

You do not need to understand every published variant to understand the root cause. The proxy and the upstream parser must agree on where one request ends and the next begins. If they do not, the attacker may be able to hide a request from one hop and reveal it to the next.

This is one more reason reverse proxies are security-critical software, not just convenience infrastructure.

When you suspect the reverse proxy, inspect the boundary directly

Troubleshooting improves dramatically once you stop treating the proxy as an invisible middlebox and start interrogating it as a full participant.

A practical first pass usually includes:

curl -vk https://app.example.eu/
curl -vk https://app.example.eu/v1/orders/42
curl -vk -H 'Host: app.example.eu' http://10.20.0.15/

Those three requests answer different questions:

• what does the public edge actually present and return?
• does routing differ by path?
• what happens if you hit a specific internal hop with the same logical host?

Then look at the real headers and timings, not the architecture diagram.

A useful proxy access log often includes:

client_ip=203.0.113.44 request_id=c3e4d2ac5f9a41b7 host=api.example.eu method=GET path=/v1/orders/42 status=200 upstream=10.20.1.21:9000 upstream_status=200 request_time=0.043 upstream_connect=0.001 upstream_header=0.010 upstream_response=0.041

That single line tells you much more than "the API was slow".

• If upstream_connect is high, the proxy may be struggling to reach or establish upstream sessions.
• If upstream_header is high, the backend likely accepted the request but was slow to start responding.
• If total request_time is high while upstream times are low, the client may be slow or the proxy may be buffering awkwardly.

Other good checks:

• compare public and upstream Host expectations
• inspect forwarded headers as the app actually receives them
• verify timeout values per route, not just globally
• check whether retries are enabled for unsafe methods
• inspect temporary-file usage for large body buffering
• look at active upstream connection counts and queue depth
• confirm whether long-lived routes are excluded from aggressive buffering or idle limits

The right level of detail depends on the stack, but the mindset is universal. The reverse proxy is not a magical cloud between client and service. It is a concrete state machine that owns protocol, policy, and often a large share of user-visible behaviour.

Reverse proxies matter because they define what the application gets to believe

A reverse proxy is the place where the public internet is turned into something an application can cope with. It terminates the outer connection, parses the request, chooses an upstream, reconstructs origin context, shields the backend from ugly connection behaviour, and often applies retries, caching, compression, and security policy on the way.

That makes it one of the most leverage-heavy components in a web stack.

It also makes it one of the easiest places to hide bad assumptions. If the proxy lies about the scheme, the app builds broken redirects. If it lies about the client IP, security policy goes crooked. If it retries the wrong request, data gets duplicated. If it buffers the wrong route, streaming behaviour disappears. If it disagrees with the backend about framing, the boundary becomes exploitable.

The practical lesson is blunt: treat the reverse proxy as part of the application's semantics, not as a detachable networking accessory. Once you do that, a lot of production behaviour that used to feel mysterious starts to look mechanical again.