How Kubernetes Scheduling Actually Works

Kubernetes scheduling looks simple from a distance. A Pod is Pending, the control plane picks a node, and a few seconds later the workload starts. That mental model is fine until a real cluster gets busy. Then a Deployment sticks in Pending even though dashboards show idle CPU. A stateful workload cannot mount a volume in the only zone where its claim ended up. A GPU job lands nowhere because the node is tainted, not because the card is missing. A high priority Pod triggers preemption and still does not start for another twenty seconds.

The problem is that "pick a node" hides a lot of machinery. The scheduler is not reading one capacity number and not doing round robin. It is maintaining a live cache of cluster state, queueing unscheduled Pods, running them through a plugin pipeline, scoring the feasible nodes, reserving the winner, possibly waiting on volume work or permit plugins, and finally writing a binding back to the API server. It is also making that decision from a very specific view of the world. It sees declared requests, labels, taints, topology, existing Pods, and volume constraints. It does not see your good intentions. It does not schedule on live CPU utilisation graphs. It does not repair a bad resource model for you.

If you want to understand why Pods land where they do, or why they do not land anywhere at all, you need the real mechanism. This post walks through it in the order the scheduler actually thinks: queue, filter, score, reserve, bind, and retry.

What Kubernetes Scheduling Is

Kubernetes scheduling is the control-plane process that chooses one node for a pending Pod by checking hard constraints, ranking feasible nodes, and then binding the Pod to the winning node in the API server.

That definition matters because it excludes a lot of things people casually attribute to the scheduler.

The scheduler does not create containers. The kubelet on the chosen node does that.

The scheduler does not start cgroups or pull images. The kubelet and container runtime do that.

The scheduler does not move a running Pod to a better node later because a nicer one became available. Normal Pods are not live-migrated.

The scheduler also does not make one perfect global cluster optimisation. It makes one placement decision at a time for one unscheduled Pod, using the state it currently has. That distinction explains many of the odd-looking decisions operators see in production. The scheduler is greedy by design. It tries to choose a good node now, not solve the whole cluster as a giant integer-programming problem.

At the API level, a schedulable Pod begins life with no .spec.nodeName. The default kube-scheduler watches for such Pods. When it decides on a node, it performs a bind, usually by updating the Pod binding through the API server so the Pod becomes associated with a specific node. From that point onward, the kubelet on that node sees a Pod assigned to it and begins admission and startup work.

That is why scheduling success and workload success are different states. A Pod can be scheduled correctly and still fail on image pull, CNI setup, CSI mount, or kubelet admission. Conversely, a Pod can stay Pending for reasons that have nothing to do with node liveness and everything to do with the scheduler being unable to prove that any node is currently feasible.

What The Scheduler Actually Sees

The scheduler does not query the entire cluster from scratch for every Pod. That would be too slow. Instead it maintains an internal cache fed by watches on the API server. Conceptually, it has a rolling snapshot of:

• nodes and their allocatable resources
• Pods already assigned to each node
• node labels and taints
• Pod requests, affinities, tolerations, topology spread rules, and priorities
• PersistentVolumeClaims, PersistentVolumes, StorageClasses, and CSI topology hints
• host port usage, volume limits, and some other placement-relevant facts

This cache is close to current, but it is still a cache. That matters when several Pods are competing for the same narrow capacity window. The scheduler deals with this by reserving resources in its own view before the bind completes, which we will get to later.

It also maintains internal queues rather than a single flat backlog. The important mental model is that not every unscheduled Pod is retried with the same urgency. Some Pods are ready to try immediately. Some have just failed and are in backoff. Some are known to be unschedulable unless cluster state changes in a relevant way.

The modern scheduling framework exposes queue-related extension points such as PreEnqueue, QueueSort, and requeue hints. Even if you never customise the scheduler, this explains a useful operational fact: a Pod that failed to schedule is not always hammered against the cluster every millisecond. Kubernetes deliberately backs off some retries so the control plane does not waste effort re-proving the same impossible result.

The default queue ordering is priority-based. Higher priority Pods are considered ahead of lower priority ones. Within similar priority levels, age and backoff state matter. This is why a critical control-plane add-on or an explicitly high-priority workload can jump ahead of a long line of ordinary application Pods.

The other crucial point is what data the scheduler trusts for capacity accounting: requests, not live usage.

If a node has 8 vCPUs and the current workloads are using only 2 in real time, that does not mean the scheduler sees 6 free CPUs. It sees allocatable - sum(requests already admitted), along with a few other accounting rules. If the running Pods have collectively requested 7.5 CPUs, the node is nearly full from the scheduler's point of view even if actual utilisation is quiet at that instant.

This is one of the most common sources of confusion in Kubernetes operations. CPU graphs show physics. Scheduling works from reservations.

The Scheduling Cycle And The Binding Cycle

Since Kubernetes 1.19, the scheduler's pluggable framework has been the cleanest way to understand what happens. Each scheduling attempt is split into two parts:

1. the scheduling cycle, which chooses a node
2. the binding cycle, which applies that choice

The scheduling cycles run serially. Binding cycles can overlap. This means the scheduler decides one Pod placement at a time, but while one Pod is being bound or waiting in a later stage, another Pod can already be in a separate binding cycle.

The framework exposes extension points. The names are worth knowing because they line up with real scheduler behaviour and with the plugin configuration you can inspect on a customised cluster.

A minimal scheduler profile can customise these stages directly:

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
  - schedulerName: default-scheduler
    plugins:
      score:
        enabled:
          - name: NodeResourcesBalancedAllocation
            weight: 1
          - name: PodTopologySpread
            weight: 2

In a default cluster you usually never touch this file, but the plugin names still matter because they tell you which part of the placement logic is active. If a platform team disables PodTopologySpread, the cluster stops preferring even replica distribution in the way many application teams assume is automatic.

One more operational wrinkle is that a cluster can run more than one scheduling profile from the same kube-scheduler process. Profiles are selected by .spec.schedulerName. Most Pods use default-scheduler, but a platform team can define another profile with different scoring weights or extra plugins and direct only certain workloads to it. That does not mean there are two independent schedulers with two independent queues. Kubernetes still requires all profiles in one scheduler instance to share the same queue-sorting plugin because there is only one pending-Pod queue underneath. In practice, this means profile-level customisation is powerful for policy differences, but it is not a clean escape hatch from global queue behaviour.

The cleanest end-to-end mental model looks like this:

1. Pick one pending Pod from the queue.
2. Build any Pod-specific state needed for the attempt.
3. Test every candidate node against hard rules.
4. If nothing fits, maybe try preemption.
5. If some nodes fit, score them.
6. Choose the highest total score.
7. Reserve the choice in the scheduler's internal state.
8. Run any wait, volume, or pre-bind work.
9. Bind the Pod to the node in the API server.
10. If anything after reservation fails, unreserve and retry later.

That reserve and unreserve pair is more important than it looks. Without it, two concurrent placements could both think the same last 2 CPUs on a node were still free.

Resource Fit Starts With Requests, Not Real Usage

The first hard gate on most clusters is resource fit. The default NodeResourcesFit plugin answers a narrow question: if this Pod were admitted on this node, would the node still have enough allocatable resource left for the Pod's declared requests?

Not limits. Requests.

If a container requests 500m CPU and 1Gi memory, that is what counts toward scheduling. A CPU limit of 2 does not make the scheduler reserve 2 CPUs. A memory limit of 2Gi does not make the scheduler reserve 2 GiB. The scheduler is deciding placement from the requested footprint.

A simple Pod example:

apiVersion: v1
kind: Pod
metadata:
  name: checkout-api
spec:
  containers:
    - name: app
      image: ghcr.io/example/checkout-api:1.4.2
      resources:
        requests:
          cpu: "700m"
          memory: "768Mi"
        limits:
          cpu: "2"
          memory: "1Gi"

From the scheduler's point of view this Pod asks for 700 millicores and 768 MiB. If the node has enough free allocatable capacity for that request, the resource-fit check may pass even if the Pod could later burst up to its CPU limit. That is a deliberate design choice. Requests are the reservation contract.

This is already more nuanced than many teams realise because the Pod request is not always just the sum of the app containers.

For each resource, the scheduler effectively accounts for:

• the sum of regular container requests
• the maximum request among init containers, because init containers run sequentially
• any Pod overhead injected through RuntimeClass
• special resources such as huge pages, GPUs, or vendor-specific extended resources

Pod overhead matters more than many people think. If a runtime class adds overhead for sandboxing, the scheduler includes that overhead in placement accounting. Kubernetes documents this explicitly for Pod Overhead. A sandboxed Pod can therefore fail to schedule on a node that seems large enough if operators are only eyeballing container requests and ignoring the extra Pod-level cost.

An example with overhead and init work:

spec:
  runtimeClassName: kata-fc
  initContainers:
    - name: migrate
      resources:
        requests:
          cpu: "1500m"
          memory: "512Mi"
  containers:
    - name: api
      resources:
        requests:
          cpu: "700m"
          memory: "768Mi"
  overhead:
    cpu: "250m"
    memory: "120Mi"

For CPU, this Pod is not treated as 700m. It is treated as max(1500m init, 700m app sum) + 250m overhead = 1750m. For memory it becomes max(512Mi init, 768Mi app sum) + 120Mi overhead = 888Mi.

That arithmetic is why short-lived init containers can still block placement on smaller nodes. They shape the feasibility test even though they do not run at the same time as the main containers.

A few other details routinely surprise operators:

Allocatable is not capacity

Nodes advertise capacity, but the scheduler uses allocatable, which is lower after kubelet reservations, system daemons, eviction thresholds, and sometimes provider-specific reservations are taken into account. If a VM has 16 GiB on paper, the scheduler may only be willing to place against 14.5 GiB.

Ephemeral storage can block placement

Pods can request ephemeral storage. If they do, NodeResourcesFit treats that as another hard fit dimension. Teams often discover this only after log-heavy workloads start staying Pending on nodes with apparently healthy CPU and memory headroom.

Extended resources behave like hard integers

A GPU request such as nvidia.com/gpu: 1 is not scored softly. A node either advertises that extended resource and has an available unit, or it does not. Device plugins make these resources visible to the scheduler, but once visible they participate in placement as strict resource requirements.

Max Pod count matters

Even if CPU and memory are available, a node may still be infeasible because it has reached its configured Pod density. On cloud clusters this often appears as "Too many pods" long before the instance type is physically exhausted.

This is why sensible requests are one of the core disciplines of operating Kubernetes well. Under-request and you get noisy neighbours and throttling. Over-request and the scheduler starts reporting artificial scarcity.

Constraints And Preferences Remove Nodes Before Ranking Starts

After basic resource accounting, the scheduler applies a second family of rules: placement constraints. These determine where a Pod is allowed to run, where it would prefer to run, and which existing placements it should stay near or far from.

The important distinction is between hard rules and soft rules.

Hard rules eliminate nodes in the filter phase. Soft rules survive into the score phase and influence ranking.

The most common mechanisms are:

• nodeSelector
• required and preferred nodeAffinity
• inter-Pod affinity and anti-affinity
• topologySpreadConstraints
• taints and tolerations
• hostPort and some volume-related restrictions

A realistic Pod can mix several of them:

apiVersion: v1
kind: Pod
metadata:
  name: payments-api
  labels:
    app: payments-api
spec:
  priorityClassName: service-critical
  nodeSelector:
    kubernetes.io/os: linux
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values: [eu-central-1a, eu-central-1b]
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 50
          preference:
            matchExpressions:
              - key: node.kubernetes.io/instance-type
                operator: In
                values: [c7i.2xlarge]
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels:
              app: payments-api
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app: payments-api
  tolerations:
    - key: dedicated
      operator: Equal
      value: payments
      effect: NoSchedule
  containers:
    - name: api
      image: ghcr.io/example/payments-api:2.1.0
      resources:
        requests:
          cpu: "500m"
          memory: "512Mi"

This Pod is saying several things at once:

• only Linux nodes are acceptable
• only two specific zones are acceptable
• it prefers a certain instance type
• it must not share a node with another payments-api Pod
• it should stay evenly spread across zones with maximum skew 1
• it may run on nodes dedicated to the payments workload because it tolerates that taint

The scheduler turns that declaration into a cascade of tests.

Node selectors and node affinity

nodeSelector is the simplest hard gate. All listed labels must match. requiredDuringSchedulingIgnoredDuringExecution is the richer version. It allows operators like In, NotIn, Exists, DoesNotExist, Gt, and Lt, and multiple selector terms.

The last part of the name matters too. IgnoredDuringExecution means Kubernetes does not evict the Pod if the node label changes later. These are placement-time rules, not continuous runtime guarantees.

Taints and tolerations

Taints push Pods away from nodes. Tolerations let specific Pods survive that push. The Kubernetes documentation is careful here: tolerating a taint does not mean the Pod will be scheduled there. It only means the taint no longer blocks that node. The scheduler still evaluates all the other constraints.

This subtlety is why GPU clusters are commonly modelled with both an extended resource request and a taint. The request says "I need a GPU". The taint says "non-GPU Pods should stay off these expensive nodes".

Inter-Pod affinity and anti-affinity

These rules compare the incoming Pod against existing Pods, not just node labels. They are powerful and expensive. Anti-affinity is often used so replicas of the same service do not land on the same host. Affinity can force related services into the same zone or node group.

This is also where people accidentally create impossible placement logic. A strict anti-affinity rule across kubernetes.io/hostname requires enough distinct nodes. A strict affinity rule to a label that only one low-priority Pod currently satisfies can also interact badly with preemption, because removing the target Pod would defeat the affinity rule itself.

Topology spread constraints

These are the cleaner modern way to express "do not pile my replicas into one failure domain". The scheduler counts matching Pods across topology domains such as zones or hostnames and tries to keep the skew within the declared limit.

whenUnsatisfiable: DoNotSchedule makes the rule hard. ScheduleAnyway turns it into a preference. maxSkew is the allowed imbalance. Kubernetes also has more advanced knobs such as nodeAffinityPolicy and nodeTaintsPolicy that affect which domains are counted.

The operational result is simple: topology spread can keep workloads resilient, but it can also make a nearly-full cluster look emptier than it really is. If a zone is saturated and the skew rule is hard, the scheduler may reject a node in the healthy zone even though it has capacity, because placing there would break the spread contract.

Scoring Chooses The Least-Wrong Feasible Node

Once hard constraints are done, the scheduler may still have several feasible nodes left. This is where scoring happens.

The score phase is often misunderstood. It is not one magic formula. It is a set of plugins that each return a score for every feasible node. Those scores are normalised into a fixed range, then combined using plugin weights. The node with the highest total wins. If several nodes tie, the scheduler picks one of the top-scoring nodes rather than pretending there is deep meaning in a perfect tie.

Default scoring plugins commonly include some mix of:

• NodeResourcesFit
• NodeResourcesBalancedAllocation
• ImageLocality
• NodeAffinity
• TaintToleration
• PodTopologySpread
• InterPodAffinity
• VolumeBinding when storage capacity scoring is enabled

It helps to think of scoring as a negotiated argument among plugins.

A node may score well because it already has the image layers cached. Another may score well because adding the Pod keeps CPU and memory more balanced. Another may score well because it improves zone spread. The scheduler adds those opinions together according to the configured weights.

Imagine a Deployment replica that is feasible on three nodes after filtering:

If spread is weighted heavily, fra-b-07 may beat fra-a-03 by one point even though the image is colder there and the raw resource fit is slightly worse. That is not an error. It is the result of the policy encoded by the enabled scoring plugins.

This is also why scheduler customisation can materially change cluster behaviour without changing any workload manifests. Platform teams sometimes tune NodeResourcesFit scoring to prefer bin packing rather than spreading. In Kubernetes terms that means changing the scoring strategy, for example from LeastAllocated to MostAllocated or RequestedToCapacityRatio. The first tends to spread headroom. The second tends to pack workloads more tightly, which may be good for cluster autoscaler efficiency but worse for failure blast radius.

A useful rule of thumb:

• filter answers "can this node run the Pod?"
• score answers "which feasible node is preferred?"

If you do not keep those two questions separate, scheduler behaviour looks random. It is not random. It is layered.

When Nothing Fits: Backoff, Unschedulable Pods, And Preemption

The most interesting scheduler behaviour starts when no node survives filtering.

At that point the Pod is unschedulable for the current cluster state. That does not always mean it will stay that way. It may become schedulable if:

• another Pod exits
• a node becomes Ready
• a volume becomes available
• a taint is removed
• cluster autoscaler adds a node
• a previously backlogged high-priority Pod finishes preempting victims

This is why the scheduler keeps unschedulable Pods in internal queues instead of dropping them on the floor.

The observable symptom is usually an event like this:

Warning  FailedScheduling  4s (x12 over 2m13s)  default-scheduler  0/8 nodes are available: 2 Insufficient cpu, 2 node(s) had untolerated taint {dedicated: payments}, 3 node(s) didn't match Pod topology spread constraints, 1 node(s) had volume node affinity conflict.

That one line is dense with information. The scheduler is telling you not just that the Pod failed, but which filters removed nodes. Read these messages literally. They are often the fastest path to root cause.

Backoff is deliberate

After failed attempts, Pods can move through backoff so the scheduler does not burn CPU endlessly retrying an impossible placement. Lower-frequency retries are normal. If a Pod fails to schedule now, that does not mean the scheduler has forgotten it. It means the Pod is being retried when cluster events make a retry worthwhile or after its backoff window expires.

Cluster autoscaler is separate

A pending Pod does not create nodes by itself. The cluster autoscaler watches for unschedulable Pods and decides whether adding nodes would help. This means a Pod can sit Pending for a while even in a healthy autoscaling cluster because the scheduler and autoscaler are two different control loops with different jobs.

Preemption happens in postFilter

If no node fits, the default scheduler may attempt preemption. The idea is straightforward: can evicting one or more lower-priority Pods from a node make enough room for the pending higher-priority Pod?

If yes, the scheduler can nominate that node and trigger victim eviction.

The Kubernetes priority and preemption documentation highlights several details that matter in real clusters:

• preemption only considers lower-priority victims
• victims get their graceful termination period
• PodDisruptionBudget is best effort, not an absolute shield
• the pending Pod gets nominatedNodeName, but may still end up elsewhere
• Kubernetes does not do cross-node preemption

That last point is underappreciated. If a Pod would fit on node A only if some conflicting Pod on node B were removed because of zone-wide anti-affinity, the default scheduler does not solve that with cross-node preemption. The node is just considered infeasible.

Preemption also does not mean immediate start. Victims may take time to terminate. During that gap, a different node may open up or a higher-priority Pod may arrive and steal the nominated spot. This is why operators sometimes see nominatedNodeName set but the final nodeName differ later.

There is also a non-preempting mode. A PriorityClass can set preemptionPolicy: Never, which means the Pod jumps ahead in queue ordering because of its priority but is not allowed to evict lower-priority Pods. This is useful for batch work that should be preferred over ordinary jobs but should not discard already-running work.

Storage Topology Turns Scheduling Into A Two-Sided Constraint Problem

Stateful workloads add a second placement system to the node-placement problem: storage topology.

If a PersistentVolume is globally reachable, this is not very dramatic. If the storage is zone-bound, node-bound, or local, the scheduler has to coordinate Pod placement with volume placement. That is what the VolumeBinding, VolumeZone, VolumeRestrictions, and NodeVolumeLimits plugins are for.

The classic trap is a topology-constrained StorageClass using immediate binding. The PVC gets bound as soon as it is created, before the scheduler has picked a node for the Pod. If the chosen volume lives in the wrong zone for the Pod's other constraints, you now have a perfectly real PVC and a Pod that can never use it.

Kubernetes has a direct mitigation for this: volumeBindingMode: WaitForFirstConsumer.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: low-latency
provisioner: csi-driver.example
volumeBindingMode: WaitForFirstConsumer

In this mode, Kubernetes delays binding or dynamic provisioning until a Pod that uses the claim exists. The scheduler can then choose a node while considering both the Pod's placement rules and the storage topology. The official storage documentation is explicit here: WaitForFirstConsumer is intended to avoid unschedulable Pods caused by topology-constrained storage being bound too early.

This is particularly important for:

• zonal block volumes
• local persistent volumes
• clusters where only some nodes run the relevant CSI driver
• high-performance storage pools reserved to specific node groups

The interaction gets even more subtle with local volumes. A local PV can only be mounted from the node that physically hosts it. At that point scheduling is no longer "choose any good node". It is "choose a node that matches the PV, the node labels, the taints, the requested resources, and the spread rules, all at once".

One more sharp edge: do not use .spec.nodeName to bypass the scheduler on workloads that rely on WaitForFirstConsumer. The storage documentation calls this out directly. Bypassing the scheduler also bypasses the volume-aware scheduling logic, and the PVC can remain Pending indefinitely.

What Happens After The Scheduler Picks A Node

Once a node wins scoring, the story is not over.

The scheduler first reserves the choice in its own state. This stops other concurrent decisions from spending the same apparent capacity. Then any permit plugins can delay the bind, and preBind plugins can do work that must succeed before the final binding is safe.

Volume handling often appears here. For example, the scheduler may need to confirm a claim can bind to a concrete volume for that node, or set up the objects that let the CSI side finish the job.

Only after those stages succeed does the bind happen. The default binder writes the scheduling decision back through the API server. At that moment the Pod has a node assignment.

Then the kubelet takes over.

The kubelet on that node performs its own admission checks, creates the Pod sandbox, configures cgroups, mounts volumes, pulls images, calls the CNI plugin for networking, and starts containers. Any of those can fail even though scheduling was correct.

This distinction is operationally important:

• Pending with no node assigned usually means a scheduling problem.
• Pending with a node assigned often means kubelet, image, network, or volume startup work is still in progress or failing.
• ContainerCreating means the scheduler is out of the picture already.

Teams sometimes stare at the scheduler when the real issue is a broken CSI driver or an image pull secret problem. The node assignment line tells you when to stop blaming placement.

How To Read A Pending Pod Like An Operator

When a Pod does not schedule, the fastest path is usually not a dashboard. It is the object itself.

Start with:

kubectl describe pod payments-api-7d7f4d8db7-4d9pf

Then read the events carefully. Scheduler events are often precise enough to tell you the real blocker in one pass.

A few common messages and what they usually mean:

If the events are not enough, inspect the node and workload declarations that feed those decisions:

kubectl get pod payments-api-7d7f4d8db7-4d9pf -o yaml
kubectl get nodes --show-labels
kubectl describe node fra-b-07
kubectl get pvc,pv,storageclass

For scheduler latency or control-plane diagnosis, the scheduler's own logs and metrics become useful. Prometheus metrics such as scheduler_pod_scheduling_duration_seconds and scheduler_framework_extension_point_duration_seconds can show whether the pain is in queueing, scoring, or a specific extension point. On very large clusters, expensive inter-Pod affinity rules or broad topology-spread calculations can show up as measurable scheduler cost.

A practical debugging sequence looks like this:

1. Is nodeName empty? If yes, stay in scheduling-land.
2. Read the latest FailedScheduling events.
3. Compare those reasons to the Pod's requests, selectors, affinities, tolerations, spread rules, priority, and PVCs.
4. Check whether autoscaler is supposed to help and whether the Pod shape actually matches a node group it can create.
5. Only after node assignment exists, switch attention to kubelet, CSI, CNI, and image-pull diagnostics.

That simple split saves a lot of wasted time.

Common Scheduling Mistakes That Look Like Cluster Capacity Problems

Many Kubernetes scheduling incidents are really modelling incidents.

Mistake 1: treating requests like documentation

Requests are not comments. They are the scheduler's reservation input. If teams copy cpu: 4 and memory: 8Gi into everything because it "feels safe", the cluster becomes artificially scarce. If they set cpu: 10m for a service that regularly burns a full core, the scheduler overpacks nodes and kubelet-level reality becomes ugly later.

Mistake 2: combining too many hard rules

A Pod that requires a specific zone, a specific instance label, strict anti-affinity, strict topology spread, a dedicated taint toleration, and a zone-bound PVC may have only one or two legal homes in the whole cluster. That is fine if intentional. It is disastrous when accidental.

Hard rules multiply. They do not add gently.

Mistake 3: using anti-affinity where topology spread would do

Strict Pod anti-affinity across hostname is often heavier and less forgiving than operators really want. If the goal is "keep replicas reasonably apart", topology spread constraints often express that more directly and can degrade more gracefully when configured as soft preferences.

Mistake 4: forgetting that taint toleration is not attraction

A toleration only removes a block. It does not tell the scheduler to prefer the node. If you want workloads to land on a dedicated pool, pair tolerations with node affinity or selectors.

Mistake 5: ignoring storage topology

When a stateful workload is stuck Pending, many teams look only at CPU and memory. A zonal or local PVC can be the real limiter. Volume placement is part of scheduling, not a separate afterthought.

Mistake 6: expecting preemption to be instant or magical

Preemption cannot break all rules. It does not do cross-node preemption. Victims terminate gracefully unless configured otherwise. A high-priority Pod can therefore stay Pending for longer than people expect even after the scheduler has decided whom to evict.

The Real Mental Model

The cleanest way to think about Kubernetes scheduling is this:

1. A pending Pod enters a priority-ordered queue.
2. The scheduler looks at declared state, not at your application's intent.
3. Hard filters remove illegal nodes.
4. Soft preferences rank the survivors.
5. The scheduler reserves a winner, completes any pre-bind work, and writes a binding.
6. The kubelet on that node then tries to make the Pod real.

That model is much more useful than the folk version of "Kubernetes just picks a node with free resources".

A Kubernetes cluster in Amsterdam, Frankfurt, or Paris can look underutilised on dashboards and still be effectively full for a particular Pod shape. The missing capacity might be CPU requests, a taint mismatch, a host-port collision, a skew rule, a PVC bound to the wrong zone, or a priority conflict that needs preemption. The scheduler is the place where all those constraints meet.

Once you understand that, Pending Pods stop feeling mysterious. They become what they actually are: proofs that the cluster, as currently declared, cannot satisfy one specific contract yet.