How gRPC Actually Works

gRPC gets described as if it were a single thing. Usually the description is "binary RPC over HTTP/2" and then the conversation moves on. That sentence is not wrong, but it is too small to explain why one slow stream can pin a backend for hours, why an HTTP 200 can still mean your call failed, why browser teams end up using gRPC-Web instead of native gRPC, or why a rollout that looks healthy in the load balancer can still leave half the traffic glued to the old fleet.

The useful mental model is that gRPC is a stack. At the top you define a contract in Protocol Buffers. In the middle you get RPC semantics such as deadlines, streaming, status codes, and generated client stubs. Underneath that, most deployments use HTTP/2 streams on one long lived connection, usually over TLS, with real transport details such as ALPN, flow control, header compression, and drain behaviour. If you only understand one layer, production behaviour looks arbitrary.

This article walks the stack from the wire upward and then back down again. We will use a warehouse service spread across Amsterdam, Frankfurt, Milan, and Dublin, because gRPC becomes much easier to reason about once you attach its channels and streams to real backends instead of treating everything as abstract middleware. We will look at how .proto contracts become method paths, how one unary call appears on the wire, why the 5 byte message envelope matters, how streaming changes backpressure, how deadlines move through the system, how load balancing really happens, and where proxies and browsers force translation.

gRPC Starts With A Contract, Not A URL Handler

A REST handler is often introduced from the outside in. There is a path, an HTTP method, maybe a JSON schema, and then the code. gRPC starts from a different centre of gravity. The primary object is the service contract.

A minimal service definition might look like this:

syntax = "proto3";
 
package inventory;
 
service Stock {
  rpc Reserve(ReserveRequest) returns (ReserveReply);
  rpc WatchLevels(WatchRequest) returns (stream StockLevel);
}
 
message ReserveRequest {
  string sku = 1;
  uint32 quantity = 2;
  string warehouse = 3;
}
 
message ReserveReply {
  string reservation_id = 1;
  uint32 reserved = 2;
}

That file does several jobs at once.

First, it defines the API surface in a language neutral way. A Go service, a Java batch worker, and a TypeScript control plane can all generate compatible clients from the same contract.

Second, it fixes the binary schema. Field numbers are not decoration. They are part of the on the wire format. Once warehouse = 3 exists in production, reusing field number 3 for some other meaning later is a schema break even if the field name changed and the code still compiles. Good protobuf hygiene is less about pretty file structure and more about never making deployed bytes mean something new.

Third, it fixes the RPC identity. The method name is not an annotation somebody might ignore later. It becomes part of the request path on the HTTP/2 hop. inventory.Stock/Reserve is not just naming. It is routing material.

Generated code then wraps that contract in the host language. The client gets a stub method like Reserve(ctx, req) or client.reserve(request). The server gets a typed handler interface it must implement. This is one of gRPC's biggest practical advantages. The contract is compiled into both sides, so the happy path is fast to use and hard to drift accidentally.

That convenience also hides sharp edges.

Schema evolution is safe only if the wire numbers stay honest

The protobuf compiler makes additive evolution look easy, and often it is. Adding a new optional field is usually fine because older readers ignore unknown fields and newer readers tolerate absent ones. The mistakes start when teams treat the file like ordinary source code instead of stable wire format.

Common safe moves:

• add a new field with a fresh number
• stop sending an old field but keep its number reserved
• add a new RPC method alongside an old one
• widen meaning in a backwards compatible way, such as allowing an extra enum value only after old readers can safely ignore it

Common dangerous moves:

• reusing a deleted field number for a new meaning
• changing field type in a way that changes wire encoding
• turning one semantic operation into another while keeping the same method name
• pushing business critical optionality into comments instead of schema shape

A lot of "gRPC broke" incidents are really protobuf contract mistakes that happened weeks earlier.

Generated stubs are transport aware even when they look simple

That friendly generated Reserve method is not a plain function call. It is a transport aware wrapper that:

• serialises the protobuf message
• chooses or reuses a channel
• opens an HTTP/2 stream on that channel
• sends metadata and body frames
• tracks deadline and cancellation state
• decodes trailers into a gRPC status

That is why gRPC feels like a local call in code and emphatically is not one in behaviour. The contract gives you type safety. It does not erase the network.

A Channel Is A Long Lived HTTP/2 Transport, Not A Single Request

The unit that matters most in production gRPC is usually the channel.

A channel is the client's long lived communication object for a target service. Different language stacks name the internal pieces slightly differently, but the same core idea shows up everywhere: resolve a name, create one or more transport connections, negotiate protocol settings, keep that state warm, and reuse it for many RPCs.

For a secure deployment, opening that first channel usually means:

1. resolve stock.internal.eu to one or more addresses
2. open a TCP connection to one selected endpoint
3. negotiate TLS
4. use ALPN to agree on h2
5. exchange HTTP/2 settings
6. start opening streams for RPCs

That sequence is why channel reuse matters so much. If a caller throws the channel away after every unary request, it loses nearly everything gRPC is trying to make cheap.

The channel carries more state than most teams realise

A reused channel is not just an open socket.

It usually carries:

• name resolution results and refresh policy
• subchannel state for candidate backends
• TLS session state
• HPACK header compression context
• peer HTTP/2 settings, including MAX_CONCURRENT_STREAMS
• connection level flow control state
• keepalive and idle timers
• health and backoff state after failures

That is why one bug in channel lifecycle code can distort latency, load balancing, and reconnect behaviour all at once. If a service opens a fresh channel per call, it creates TLS churn, loses multiplexing, defeats sticky header compression gains, and can cause very bursty backend connection patterns during traffic spikes.

One channel does not always mean one TCP connection forever

Language runtimes differ in how they manage subchannels, connection pools, and resolver updates. Some maintain several ready transports underneath the high level channel object when load balancing policy calls for it. Some open a new connection when the old one becomes unhealthy or when the server sends GOAWAY. The important point is not the exact class diagram in one library. The important point is that RPCs are normally scheduled onto already managed transport state, not created from zero each time.

This distinction matters when people ask, "Why did adding three new pods not rebalance immediately?" The answer is often that the clients already had healthy channels and saw no reason to reconnect. Discovery can know about the new backends while most actual calls continue on old warm subchannels.

HTTP/2 is not an implementation detail for gRPC

Teams sometimes talk as if gRPC "uses HTTP/2 under the hood" in the same weak sense that an ORM uses TCP under the hood. That undersells the relationship. gRPC's ordinary semantics depend directly on HTTP/2 features:

• stream multiplexing
• binary frame boundaries
• flow control on streams and connections
• trailing headers
• graceful drain with GOAWAY
• one connection carrying many concurrent calls

If an intermediary downgrades that hop to HTTP/1.1 without explicit translation, native gRPC stops working because the semantic substrate disappeared.

This is also why some proxy settings that look harmless on REST paths become fatal on gRPC paths. An idle timeout that is generous for short JSON requests may kill a healthy bidirectional stream. Buffering that is fine for upload forms may destroy streaming response behaviour. Header limits that rarely matter for browser traffic may choke on large metadata or identity headers.

A Unary RPC On The Wire Is Headers, One Message Envelope, And Trailers

The cleanest way to understand gRPC is to follow one unary call on the wire.

Suppose a client in Amsterdam calls Reserve on the inventory.Stock service.

The request is typically an HTTP/2 POST whose path identifies the service and method:

:method: POST
:scheme: https
:authority: stock.internal.eu
:path: /inventory.Stock/Reserve
content-type: application/grpc
te: trailers
grpc-timeout: 180m
grpc-accept-encoding: identity, gzip

Several details matter here.

POST is the ordinary method because the semantics are RPC, not generic resource retrieval.

The method path is not /reserve-stock or some application router string. It is /package.Service/Method. That exact shape is what many proxies, observability filters, and service meshes key on for routing and policy.

content-type: application/grpc tells the peer this is native gRPC framing.

te: trailers looks quaint, but it matters. gRPC relies on trailing headers for final status. Historically this header helped ensure intermediaries treated trailers correctly.

grpc-timeout carries the remaining deadline budget as a relative duration. 180m means 180 milliseconds, not 180 minutes. The allowed units are defined by gRPC's wire rules, not by generic HTTP duration conventions.

After the request headers, the client sends the message body. gRPC does not drop the protobuf bytes straight into the stream. Each message gets a small envelope:

1 byte   compressed flag
4 bytes  message length, big-endian
N bytes  message payload

If the protobuf body is 26 bytes, the first five bytes look like this:

00 00 00 00 1A

That means:

• 00: this message is not using per-message compression
• 00 00 00 1A: the payload length is hexadecimal 0x1A, which is 26 bytes

Only after that does the actual protobuf message begin.

Protobuf bytes are compact because the field numbers drive the encoding

Take this message:

message ReserveRequest {
  string sku = 1;
  uint32 quantity = 2;
  string warehouse = 3;
}

If the request is:

• sku = "PUMP-9"
• quantity = 12
• warehouse = "Rotterdam"

then a representative protobuf payload is:

0A 06 50 55 4D 50 2D 39 10 0C 1A 09 52 6F 74 74 65 72 64 61 6D

You do not need to memorise the bytes, but understanding the shape helps:

• 0A is field 1 with wire type length-delimited
• 06 is the string length for PUMP-9
• 10 is field 2 with varint encoding
• 0C is decimal 12
• 1A is field 3 with wire type length-delimited
• 09 is the string length for Rotterdam

This is why small protobuf messages stay lean compared with verbose JSON. The schema is carried mostly by field numbers and types known to both sides, not by repeating full key names in every payload.

The response is not finished when the data message arrives

A successful response usually starts with HTTP/2 headers such as:

:status: 200
content-type: application/grpc

Then the server sends one or more data frames containing the gRPC message envelope plus protobuf bytes.

The final RPC outcome arrives in trailing headers:

grpc-status: 0

If the call failed at the application layer, the HTTP status may still be 200 while the trailers say something like:

grpc-status: 14
grpc-message: upstream unavailable

That behaviour surprises people coming from conventional HTTP APIs. In gRPC, HTTP status tells you whether the HTTP exchange itself stayed valid enough to carry the RPC protocol. The RPC status tells you whether the call succeeded as a gRPC operation.

This is one reason dashboards built only around HTTP status codes give misleading comfort. A proxy may happily report 200 responses while the application layer is returning UNAVAILABLE or DEADLINE_EXCEEDED in trailers.

Protobuf Encoding Keeps Messages Small, But It Also Freezes Important Design Choices

People often talk about protobuf as if its only job were saving bandwidth. The bandwidth win is real. It is also the least interesting part after the first few months.

What matters more is that protobuf forces you to make wire level decisions early.

Field numbers become part of the public interface

In ordinary source code you can rename fields freely if you update callers. In protobuf the human readable name is the least important part. Field number and wire type are what deployed readers and writers actually exchange.

That is why responsible schema evolution includes rules such as:

• reserve deleted field numbers so they are never reused
• prefer additive change over in place semantic replacement
• treat oneof changes carefully because they alter meaning, not just syntax
• document default value assumptions explicitly

Ignoring those rules creates failures that are very hard to diagnose later because both sides may still parse the payload successfully while disagreeing about what it means.

Compact encoding changes performance shape, not only payload size

A 1 KiB JSON document and a 300 byte protobuf message do not just differ in network usage. They also differ in:

• parsing cost
• allocation behaviour
• cache footprint
• serialization overhead on high call rates

That does not make protobuf automatically better for every workload. It does make it attractive for high volume service to service calls where the schema is stable and the call rate is high enough that repeated textual encoding becomes real CPU cost.

Generated types are only as good as the schema discipline behind them

A generated client stub feels safe because the method signature is typed. That safety can be false if the protobuf schema is poorly designed.

Examples:

• a field is marked optional in practice but the business logic treats it as mandatory
• an enum has an UNKNOWN = 0 default but handlers forget to reject it
• a huge bytes field silently becomes the hot path for multi-megabyte uploads
• two semantically distinct operations share one message type and become impossible to evolve independently

The failure is not in gRPC. The failure is that the schema became an under-designed application protocol.

Message size limits are real operational constraints

Many gRPC libraries ship with conservative inbound message limits, often a few mebibytes unless configured otherwise. That is not paranoia. A single stream carrying a very large message has to allocate buffers, consume flow-control credit, and potentially block other work on the same connection.

If an API routinely needs 20 MiB objects, the better design may be:

• send metadata and small control messages over gRPC
• move bulk objects through object storage or chunked streaming
• keep the RPC responsible for coordination rather than raw blob transfer

gRPC is efficient, but it is not a licence to forget that bytes still occupy memory and queues.

Streaming RPCs Change The Ownership Of Time And Backpressure

Unary calls are the easy case. Streaming is where gRPC starts exposing the transport details many teams hoped not to think about.

There are four standard RPC shapes:

All four shapes still map one RPC to one HTTP/2 stream. The difference is how long that stream stays open and how many message envelopes travel across it.

Streaming moves the critical resource from requests per second to bytes in flight over time

A unary call can often be reasoned about as a quick transaction. A bidirectional stream might live for minutes or hours. That changes almost every operational question.

Instead of only asking "how many calls per second?", you start asking:

• how many open streams per channel?
• how much buffered data per stream?
• how much flow-control credit is outstanding?
• how long can an idle stream remain healthy through proxies and load balancers?
• what happens if the consumer reads slowly or stops entirely?

A price feed from Frankfurt to Madrid might keep one stream open all trading day. That is efficient. It also means a deployment, load balancer policy change, or proxy idle timeout now interacts with a connection that carries durable business meaning, not just a short request.

Half-close behaviour matters

In client streaming and bidirectional calls, "done sending" and "done receiving" are separate moments.

The client can finish its outbound messages and half-close the stream while still waiting for the server's reply. The server can likewise stop sending and still accept inbound frames until the stream is fully closed. If application code gets this wrong, streams sit open unnecessarily, leak memory, or deadlock waiting for a side that already finished logically.

Streaming does not remove the need for application level sequencing

Because gRPC preserves message boundaries, teams sometimes assume application semantics are solved automatically. They are not.

A stream still needs clear answers to questions such as:

• is every message independently idempotent?
• how does the receiver detect duplicates after reconnect?
• can messages arrive after the business session is already obsolete?
• what marks the last useful update?
• how is version skew handled mid stream?

The transport gives you ordered messages on one stream. It does not define your session protocol for you.

Flow Control, Deadlines, And Cancellation Are The Real Latency Controls

A lot of gRPC performance talk focuses on serialization speed. In production, the more decisive controls are often flow control, deadlines, and cancellation.

HTTP/2 flow control is the backpressure mechanism under gRPC

Every HTTP/2 connection has a connection level window. Every stream also has its own window. A sender can transmit only while it has credit. The receiver returns credit with WINDOW_UPDATE as it reads and processes bytes.

This matters immediately for streaming workloads.

Suppose a client in Milan opens a server streaming RPC and each update is 96 KiB after gRPC framing. If the per-stream window is 64 KiB, one full message cannot complete in a single push. The sender has to stop partway through and wait for more credit. That pause is not a bug. It is the protocol preventing unbounded buffering into a consumer that has not kept up.

If several active streams share one connection, the connection window becomes the next limit. One noisy stream can consume a large share of available connection credit, especially if application code reads one stream aggressively and neglects others.

Deadlines are not decorative timeout settings

gRPC treats deadlines as part of the RPC contract.

A client computes how much time the call may spend and propagates that budget on the wire, commonly through grpc-timeout. Servers should stop pointless work once the call can no longer succeed in time. Downstream gRPC clients should propagate the reduced remaining budget rather than inventing a fresh full timeout.

That behaviour matters because queues compose. If a frontend gives itself 250 ms, then waits 220 ms before calling a backend with a brand new 500 ms timeout, the system has already lied to itself. The original user budget is gone. Honest deadline propagation prevents work that is guaranteed to miss its caller's expectations.

Cancellation is normal, not exceptional

In gRPC, stream cancellation happens for ordinary reasons:

• caller deadline expired
• user closed the page or disconnected the client
• higher layer logic no longer wants the result
• deploy drain is shutting down work cleanly
• one side detected the session is obsolete

Under HTTP/2 this often surfaces as RST_STREAM for the affected stream. That is not necessarily transport damage. It may be the correct way to end just one RPC while preserving the rest of the channel.

Where teams get into trouble is failing to honour cancellation promptly. If the client gave up but the server keeps scanning a huge result set, buffering messages no one will read, the system wastes CPU, memory, and queue time on work that has already lost its consumer.

Keepalive and idle policies must match the real traffic shape

Long lived but quiet streams, or warm channels with long gaps between unary bursts, expose hidden infrastructure defaults.

A reverse proxy may decide a connection idle for 60 seconds is dead and close it. A cloud load balancer may tolerate one shape of keepalive traffic but not another. A client may send aggressive pings and trigger rate limiting or server side connection policies.

This is why gRPC tuning is never just application code. Transport timers across client, proxy, and server all have to agree on what a healthy quiet connection looks like.

Load Balancing Happens At The Channel And Subchannel Level, Not Magically Per Call

This is the part many teams learn only after their first awkward scale-out.

In a plain HTTP/1.1 world, you can imagine each request as a new balancing opportunity because clients open many short lived connections or the proxy sees many discrete requests. gRPC channels change the picture. A client may resolve several backend addresses, but still keep using one already healthy subchannel for a long time.

Resolver plus load-balancing policy decides the candidate set and selection rule

A typical gRPC client stack separates two jobs:

1. resolver: turn a target name into one or more addresses, and sometimes deliver service config
2. load-balancing policy: choose which ready subchannel should receive a new RPC

Common policies include pick_first and round_robin.

pick_first does exactly what it sounds like. It tries addresses until one becomes ready, then sends new RPCs there until something forces a change.

round_robin keeps several ready subchannels and rotates new RPCs across them.

Neither policy rewrites the laws of time.

If a bidirectional stream starts on Amsterdam and runs for three hours, round robin does not migrate that live stream to Frankfurt midway through just because a new pod appeared. The choice applies when the RPC begins. Open work remains attached to the chosen subchannel.

Service config can carry balancing and retry intent

A representative service config can look like this:

{
  "loadBalancingConfig": [{ "round_robin": {} }],
  "methodConfig": [{
    "name": [{ "service": "inventory.Stock" }],
    "timeout": "0.200s"
  }]
}

Different runtimes support different subsets and surrounding delivery mechanisms, but the design idea is important. gRPC control information can travel alongside discovery, not only inside application code.

This is powerful because operators can influence call behaviour centrally. It is also dangerous if teams assume every language or proxy honours the exact same knobs the same way. Cross-language fleets need explicit validation, not wishful symmetry.

Deploy drains rely on GOAWAY, not on hope

When a server wants to stop taking fresh streams but let existing ones finish, it sends GOAWAY. The frame carries the last stream ID the sender will process. Streams with lower or equal IDs may continue. Later IDs must move elsewhere.

That enables graceful rotation:

• Amsterdam serves old streams until they finish
• Frankfurt becomes the target for new streams
• the client opens or reuses a different healthy subchannel
• the old connection retires only after in-flight work completes or times out

If you skip this and simply kill the process or let the proxy cut the connection, long lived streams fail abruptly and the client has to reconnect under pressure.

L4 and L7 layers see different things

An L4 balancer sees TCP connections. A gRPC client may place dozens of concurrent RPCs and hours of stream lifetime inside one such connection. An L7 proxy that understands HTTP/2 can reason about the individual streams and sometimes route or observe them more intelligently.

That distinction explains a lot of confusing graphs. Connection counts may look balanced while actual stream counts are skewed. One backend can carry the hot long lived channels while another gets a pile of short unary calls that come and go quickly.

Health checks are RPCs too, and they see more than a TCP probe

This is another place where gRPC's layering matters. A TCP health check can prove that a process accepted a socket. It cannot prove that the server can parse HTTP/2, decode metadata, dispatch /grpc.health.v1.Health/Check, talk to its dependencies, and return trailers correctly.

That is why serious gRPC deployments often use the standard gRPC health checking service or an equivalent application level readiness call. A backend may be reachable at the transport layer and still be a bad target because:

• it is draining and should not take new work
• its dependency in Milan is failing so only some methods work
• its thread pools are saturated even though accept still succeeds
• it can answer unary probes but not sustain long lived streams

When the balancer, the service, and the client disagree about health, the traffic pattern gets strange very quickly. A realistic health signal is not just "port open". It is "this peer can still serve the class of RPCs I am about to send it".

Proxies, Browsers, And gRPC-Web Turn Native gRPC Into A Translation Problem

Inside a service mesh or data centre, native gRPC is straightforward enough: client and server both speak HTTP/2 and both understand trailers and stream semantics. The moment a browser or a generic web edge gets involved, the picture changes.

Browsers do not expose portable native gRPC semantics cleanly enough

Web applications have fetch, WebSocket, and other high level browser APIs. They do not have a portable raw HTTP/2 client API that gives general JavaScript code the same direct control over streams, trailers, and transport behaviour that native gRPC libraries expect.

That is why browser teams often use gRPC-Web or similar translation layers.

The broad pattern is:

1. browser sends a browser-friendly request format
2. edge proxy such as Envoy translates it
3. backend service receives native gRPC over HTTP/2

The details vary, but the operational lesson is stable: browser gRPC is usually a proxy mediated protocol boundary, not an end to end native gRPC session.

Proxies have to preserve streaming semantics intentionally

A reverse proxy that works beautifully for short REST calls can quietly break gRPC if it:

• downgrades the upstream hop to HTTP/1.1 without translation
• buffers streaming responses
• enforces short idle timers
• limits header size too aggressively for metadata heavy calls
• fails to expose or forward trailing status correctly

A lot of "random gRPC disconnects" are really ordinary proxy defaults meeting long lived stream traffic for the first time.

Browser translation changes observability too

When the public edge speaks gRPC-Web and the origin speaks native gRPC, you effectively have two protocol segments:

• browser to edge
• edge to backend

Errors can originate in either segment. A browser visible failure may be a translation problem at the edge, not a backend application issue. If observability covers only the backend service, operators can spend hours debugging the wrong hop.

The rule here is simple. Every translation boundary is a new failure domain. Treat it that way.

Retries And Status Codes Work Only When Idempotency Is Explicit

gRPC's status model is clean, but using it well requires more discipline than many teams expect.

A few common status codes carry a lot of operational meaning:

• 0 OK
• 1 CANCELLED
• 4 DEADLINE_EXCEEDED
• 5 NOT_FOUND
• 8 RESOURCE_EXHAUSTED
• 13 INTERNAL
• 14 UNAVAILABLE

The temptation is to map these directly to retry behaviour. That is too simplistic.

UNAVAILABLE is retryable only when the operation itself is safe to repeat

A unary read that failed before the server committed any result is often fine to retry. A payment capture or inventory reservation may not be. gRPC gives you transport and status tools. It does not decide business idempotency for you.

That is why robust designs combine gRPC with explicit idempotency strategy:

• idempotency keys for side effecting methods
• clear distinction between safe reads and mutating writes
• server behaviour that recognises repeated requests cleanly
• retry budgets that stop storms during partial outages

Without that discipline, automatic retries simply convert one backend hiccup into duplicate business actions.

Retry support varies by runtime and by when the failure happened

Native gRPC retries generally work best before response headers are committed, because before that point the client can reasonably conclude the call did not produce an observable reply. Once headers or streaming messages are already flowing, retrying the same RPC may be impossible or unsafe.

This is another place where generated stubs make things look simpler than they are. The stub may expose one method call. Underneath, transport state has to answer a much harder question: has enough of the operation happened that retry would be ambiguous?

Status codes are a control plane for observability

If one cluster suddenly shows a rise in:

• DEADLINE_EXCEEDED, think latency, queueing, or bad timeout budgets
• UNAVAILABLE, think transport or backend availability
• RESOURCE_EXHAUSTED, think concurrency caps, message size limits, or quotas
• CANCELLED, think caller behaviour, deploy drains, or upstream abandonment

A flat error rate graph hides those differences. gRPC status families expose them directly if you record them.

Real gRPC Debugging Means Looking At Streams, Trailers, And Channel State

When a gRPC system misbehaves, the first wrong move is often to debug it like a plain REST API.

Socket counts are not enough. HTTP codes are not enough. Average latency is not enough. One warm channel can hide hundreds of independent call outcomes, mixed stream types, and a drain event that affects only future streams.

Useful gRPC observability usually includes:

• method path such as /inventory.Stock/Reserve
• gRPC status code
• deadline budget and actual duration
• message counts and bytes per direction
• active streams per channel
• resolver and subchannel state
• connection age and drain events
• cancellation reason where available

A log line that actually helps might look like this:

ts=2026-06-17T09:14:51Z
peer=10.42.7.13:8443
method=/inventory.Stock/Reserve
grpc_status=4
http_status=200
deadline_ms=180
duration_ms=186
stream_id=73
outbound_bytes=247
inbound_bytes=0
subchannel=amsterdam-2

That single record tells a much more truthful story than "request failed".

Common production failures line up cleanly once you know the layers

If a call rate spikes and only a subset of backends get hot, ask whether long lived channels are pinned and whether the balancing policy is effectively pick_first in practice.

If browser users fail while internal batch workers are fine, ask whether the gRPC-Web translation layer is mishandling trailers, idle timers, or CORS rather than blaming protobuf itself.

If streams die at suspiciously round intervals such as 60 seconds or 300 seconds, inspect proxy and load balancer idle defaults before rewriting client code.

If memory climbs during a streaming incident, check whether flow-control credit is being consumed faster than receivers read, or whether application code keeps buffering messages after cancellation.

If 200 responses rise while the product is obviously failing, inspect grpc-status in trailers before celebrating transport success.

Wire level captures are still worth knowing how to read

Tools such as Wireshark, Envoy access logs, OpenTelemetry spans, or language specific channel tracing become much more useful when you recognise the protocol shape.

You should be able to answer practical questions like:

• did the client negotiate h2 successfully?
• did the request path match the expected service and method?
• did the server send grpc-status trailers?
• was GOAWAY sent before the deploy cut traffic?
• are resets affecting one stream or the whole connection?
• is the connection window the bottleneck or the application handler?

That is the level where gRPC stops being mysterious. The protocol is not doing magic. It is just using a transport stack with more visible moving parts than a plain JSON POST.

gRPC Is Fast When The Whole Stack Agrees On What The Call Means

gRPC works well because several pieces line up cleanly.

The protobuf contract gives both sides one stable schema. HTTP/2 gives the transport stream boundaries, multiplexing, backpressure, and graceful drain. The RPC layer adds deadlines, generated stubs, status codes, and metadata. When those layers are understood together, you get a system that is efficient, typed, and operationally predictable.

When they are not understood together, teams blame the wrong layer. They blame protobuf for a load-balancing problem, blame HTTP/2 for an idempotency mistake, blame the client stub for a proxy idle timeout, or blame the service for an error that lived entirely in gRPC-Web translation.

The practical takeaway is simple. Treat gRPC as a transport stack with a contract on top, not as a magic function call library. If you can reason about channels, streams, trailers, flow control, deadlines, and drain behaviour, most production surprises stop being surprises.