How cgroups Actually Work

If you work around containers, systemd services, or Kubernetes nodes long enough, you will eventually hit a machine that looks healthy from ten metres away and broken up close.

The host still has free CPU, yet one service is getting throttled. The box still has RAM, yet one container is being OOM-killed. A workload forks happily in staging and suddenly gets EAGAIN in production. A platform team says "the pod has a 500m CPU limit" as if that were some abstract orchestration fact, when the concrete reality on the node is just a set of numbers written into files under /sys/fs/cgroup.

That gap between the friendly abstraction and the kernel mechanism is where most cgroup confusion lives.

A cgroup is not a container. It is not a namespace. It is not a scheduler by itself. It is a kernel mechanism for grouping tasks into a hierarchy, attaching resource controllers to parts of that hierarchy, and then accounting for or constraining what those tasks consume. Modern Linux uses cgroup v2 for this: one unified tree, one membership path per task, and a set of controller files that the kernel consults while making scheduling, reclaim, IO, and admission decisions.

If you want the useful mental model, think of cgroups as a live policy tree for process groups. The tree is visible as a filesystem, but the files are really kernel control points. Writing 50000 100000 to cpu.max is not editing configuration in the abstract. It is changing the budget the scheduler will enforce for every runnable task in that group. Writing 1G to memory.max is not decorating metadata. It is changing the arena within which reclaim and memcg OOM now happen.

This post walks through that mechanism in the order that matters operationally: the tree, controller enablement, CPU control, memory control, the other major controllers, systemd and Kubernetes integration, delegation rules, and the failure modes that keep surprising people.

What cgroups Are

Linux cgroups are a hierarchical resource-control system that groups processes and threads, then applies accounting and limits to those groups instead of only to individual tasks.

That definition is worth reading slowly because every word matters.

Hierarchical means a cgroup lives in a tree. A child inherits the reality of its ancestors. If a parent is limited to 2 GiB of memory, no descendant can escape that ceiling by setting a larger memory.max locally.

Resource control means the kernel uses these groups when making decisions about CPU time, memory reclaim, block IO, process creation, CPU placement, and a few other classes of resource.

Groups of tasks means cgroups are about sets of tasks, not about one PID in isolation. A service with fifty worker threads is governed as a unit. That is the crucial difference from older per-process knobs such as nice, ulimit, or setrlimit.

Accounting and limits means there are two jobs here. Sometimes you want a hard cap. Sometimes you want accurate usage, pressure, and event counters even if you do not cap anything. Both matter.

It also helps to clear up what cgroups are not.

They are not a security boundary. Namespaces, capabilities, seccomp, LSM policy, and ordinary permissions do security work. Cgroups stop noisy neighbours and provide accounting. They do not stop a process from seeing the wrong filesystem or calling the wrong syscall.

They are not just for containers. systemd uses them for ordinary host services. Desktop sessions use them. Batch schedulers use them. Database clusters use them. You can use cgroups on a host that has never seen Docker.

They are not optional background plumbing in modern Linux. If you run a distribution with systemd, you are already standing inside a cgroup tree whether you think about it or not.

The main version split looks like this:

You still need to recognise v1 names because old blog posts, old container runtime flags, and some older enterprise fleets still expose them. But if you are learning the mechanism in order to reason about current Linux behaviour, v2 is the right primary model.

Why Linux Needed Group Resource Control

Before cgroups, Linux had plenty of process-level controls. You could nice a task. You could set rlimits. You could pin something to a CPU with taskset. You could even place a whole workload under a service manager that restarted it when it died.

What you could not do cleanly was say:

• this whole web service may use at most half a core on average
• this queue worker pool may reclaim hard once it passes 768 MiB and die locally at 1 GiB
• this build job may create at most 64 tasks total
• this database slice must keep some memory protected from generic reclaim pressure caused by adjacent work

Per-process policy breaks down fast once one logical workload becomes many tasks. A threaded JVM, a prefork web server, a PostgreSQL cluster, or a container with several busy threads does not behave like one process with one obvious PID to tune.

CPU fairness is the cleanest example.

Suppose two services share one CPU core:

• api.service has 2 worker threads
• batch.service has 40 worker threads

If fairness exists only per task, the batch service can dominate CPU simply by having more runnable tasks. That may still be fair to the scheduler's internal units, but it is not fair to the operator's mental model of services.

Cgroups solve that by giving the scheduler a group concept. CPU can be divided between sibling cgroups first, then divided among the tasks inside each cgroup.

Memory needed the same shift. Without memory cgroups, a single greedy workload could push the whole machine into global reclaim and finally into a host-wide OOM decision. With memcg, reclaim and OOM can happen inside the offending group while the rest of the machine keeps breathing. That is the reason an application inside a container can die even when node-level graphs still show free memory.

IO and process counts needed the same treatment. Fork bombs are not interesting because one PID misbehaved. They are interesting because a service boundary failed. Storage contention is rarely about one thread. It is about one workload swamping the device queue.

This is why cgroups ended up underneath both containers and service managers. Once Linux gained a robust way to express resource policy for workload groups, everything above it started depending on that mechanism.

The cgroup v2 Tree Is the API

The first practical thing to understand is that cgroup v2 presents itself as a filesystem mounted at /sys/fs/cgroup.

That description is true and slightly misleading.

It is true because you really do create directories, read files, and write values. It is misleading because these files are not passive configuration blobs. They are an API surface backed by live kernel state.

A minimal tree might look like this:

/sys/fs/cgroup
├── cgroup.controllers
├── cgroup.procs
├── cgroup.subtree_control
├── system.slice
│   ├── nginx.service
│   └── postgresql.service
├── user.slice
└── workloads
    ├── api
    └── batch

Every directory is a cgroup. Every task belongs to exactly one cgroup in this hierarchy.

There are two file families to keep in mind.

The first family is generic cgroup metadata:

• cgroup.procs: PIDs in this cgroup
• cgroup.threads: thread IDs in this cgroup, mainly relevant for threaded mode
• cgroup.controllers: which controllers this cgroup could distribute to children
• cgroup.subtree_control: which controllers this cgroup has actually enabled for its children
• cgroup.events: state such as whether the cgroup is populated
• cgroup.freeze: administrative freeze switch
• cgroup.kill: kill all descendant tasks in the cgroup
• cgroup.type: domain or threaded mode information

The second family is controller-specific files such as:

• cpu.weight, cpu.max, cpu.stat
• memory.current, memory.high, memory.max, memory.events, memory.stat
• io.stat, io.weight, io.max
• pids.current, pids.max
• cpuset.cpus, cpuset.mems

The membership operation is brutally simple. To move a process into a cgroup, write its PID into cgroup.procs:

echo 18422 | sudo tee /sys/fs/cgroup/workloads/api/cgroup.procs

From that moment, the task and any future children it forks appear under that cgroup unless moved again. That inheritance point matters. Cgroups are usually applied near the top of a service lifecycle so everything the service later spawns lands inside the correct accounting boundary automatically.

You can always inspect a task's current cgroup path through /proc:

cat /proc/18422/cgroup

On a pure v2 system you typically see a single line such as:

0::/system.slice/api.service

That line is the kernel saying: this task's unified cgroup membership path is /system.slice/api.service.

One immediate consequence follows from the tree model: every limit is contextual. The effective budget for a leaf is the tightest relevant policy on the path from root to leaf. If a parent says memory.max = 2G and the child says memory.max = 3G, the child does not get 3 GiB. The parent already narrowed reality to 2 GiB.

Controllers Do Not Flow Down Automatically

This is the first cgroup rule that bites people who try manual setups.

Just because a controller exists on the system does not mean a child cgroup automatically gets to use it.

In cgroup v2, controllers are enabled top down. A parent advertises which controllers are available through cgroup.controllers, and then explicitly turns selected ones on for its children by writing +controller tokens into cgroup.subtree_control.

A small manual example makes the rule concrete:

sudo mkdir /sys/fs/cgroup/workloads
cat /sys/fs/cgroup/workloads/cgroup.controllers
# cpu memory io pids cpuset
 
printf '+cpu +memory +pids\n' | sudo tee /sys/fs/cgroup/workloads/cgroup.subtree_control
sudo mkdir /sys/fs/cgroup/workloads/api
sudo mkdir /sys/fs/cgroup/workloads/batch

At this point api and batch can expose cpu.*, memory.*, and pids.* files because their parent chose to distribute those controllers downward.

Without that cgroup.subtree_control write, the children exist, but many controller files you expect simply are not usable there.

This design is not bureaucracy. It is what makes safe delegation possible. A parent decides which resource domains children are allowed to manage. A delegated subtree can only work with controllers its ancestor explicitly handed down.

The second rule that surprises people is the no internal process constraint for domain controllers. Outside the root cgroup, you cannot usually have both:

1. live processes sitting directly in a cgroup
2. domain controllers such as CPU or memory distributed to that cgroup's children

In other words, a cgroup generally has to choose whether it is a place where tasks live or a place where resource domains are split further.

That is why well-structured trees often look like this:

workloads/
├── api/
│   └── workers/
└── batch/
    └── jobs/

The parent workloads enables controllers for children. The leaf cgroups hold the actual tasks. If you ignore this rule, writes that look innocent start failing with EBUSY, EINVAL, or EOPNOTSUPP, and the tree feels irrational until you remember the hierarchy semantics.

This one rule explains a surprising amount of systemd's cgroup structure. Slices tend to be interior organisational nodes. Services and scopes tend to be leaves where tasks actually run.

A Small Manual Session Makes the Mechanism Concrete

You do not need Docker, Kubernetes, or systemd abstractions to see the core behaviour. A throwaway subtree and one shell are enough.

On a test machine, the sequence looks like this:

sudo mkdir /sys/fs/cgroup/demo
printf '+cpu +memory +pids\n' | sudo tee /sys/fs/cgroup/demo/cgroup.subtree_control
 
sudo mkdir /sys/fs/cgroup/demo/shell
echo '200' | sudo tee /sys/fs/cgroup/demo/shell/cpu.weight
echo '50000 100000' | sudo tee /sys/fs/cgroup/demo/shell/cpu.max
echo '768M' | sudo tee /sys/fs/cgroup/demo/shell/memory.high
echo '1G' | sudo tee /sys/fs/cgroup/demo/shell/memory.max
echo '64' | sudo tee /sys/fs/cgroup/demo/shell/pids.max
 
echo $$ | sudo tee /sys/fs/cgroup/demo/shell/cgroup.procs

The last line moves your current shell into the leaf cgroup. From that moment, every command you launch from that shell inherits the same cgroup membership unless something else moves it later.

Now the files stop looking abstract.

If you run a CPU burner from that shell, cpu.stat starts showing whether the quota is biting:

cat /sys/fs/cgroup/demo/shell/cpu.stat

If you allocate memory past the high threshold, memory.events starts counting pressure before any kill occurs:

cat /sys/fs/cgroup/demo/shell/memory.events
cat /sys/fs/cgroup/demo/shell/memory.pressure

If a buggy loop keeps creating threads, the task limit becomes visible here:

cat /sys/fs/cgroup/demo/shell/pids.current
cat /sys/fs/cgroup/demo/shell/pids.max

The important lesson from this tiny exercise is that the cgroup is not some passive label attached to a process after the fact. The kernel consults these files while the workload is running. Scheduling, reclaim, throttling, and admission decisions are all reading this policy live.

It also becomes obvious why leaf placement matters. If you had tried to keep the shell directly in demo/ while also distributing CPU and memory controllers to children, the kernel would reject that arrangement because demo/ would be trying to act as both a populated workload cgroup and an interior resource domain.

Do not keep a manual test subtree around casually on a production host. The point of the exercise is to make the model tactile: create a parent, hand controllers down, put tasks in leaves, and then read the counters that correspond to the symptom you induced.

CPU Control Means Shares and Budgets, Not Just Priorities

The CPU controller in cgroup v2 gives you two very different levers:

• cpu.weight for proportional sharing under contention
• cpu.max for a hard time budget per period

These are related, but they solve different problems.

cpu.weight decides relative share under contention

cpu.weight is the modern v2 replacement for v1's cpu.shares. Its range is 1 to 10000, and the default is 100.

If two sibling cgroups both have runnable work and share the same parent CPU domain, the scheduler uses their weights to divide CPU time proportionally.

Suppose api.service has weight 200 and batch.service has weight 100. If both want all of one CPU, the rough split is:

• API: 200 / 300 = 66.7%
• Batch: 100 / 300 = 33.3%

That rule applies only when there is actual contention. If batch.service is idle, api.service can use the whole CPU regardless of its weight. Weight is a contention rule, not a cap.

This is one place where cgroups interact directly with the scheduler internals described in any serious article on CFS group scheduling. The scheduler is not just comparing tasks. It is also maintaining runnable entities for cgroups, so fairness can exist between groups first and within groups second.

A simple configuration looks like this:

echo 200 | sudo tee /sys/fs/cgroup/workloads/api/cpu.weight
echo 100 | sudo tee /sys/fs/cgroup/workloads/batch/cpu.weight

On a busy host, that can be the right tool when you want one class of work to win more often without imposing an absolute ceiling.

cpu.max is a hard bandwidth budget

cpu.max is different. Its format is:

<quota> <period>

For example:

echo '50000 100000' | sudo tee /sys/fs/cgroup/workloads/api/cpu.max

That means: this cgroup may consume 50000 microseconds of CPU time during each 100000 microsecond period. In plain English, it gets half a core on average.

If the value is:

max 100000

then there is no quota cap, only the period value.

The important operational point is that cpu.max still bites even if the machine is otherwise idle. Weight says who wins when neighbours are hungry. Quota says you may not consume more than this budget during the period, full stop.

That is why CPU limits in container platforms often cause latency spikes that surprise people. A service might be running on a node with several idle cores, yet still get throttled because its cgroup exhausted its budget early in the current period.

A typical cpu.stat makes this visible:

usage_usec 8421931
user_usec 7010022
system_usec 1411909
nr_periods 913
nr_throttled 148
throttled_usec 22844571

The last two lines are the giveaway. nr_throttled counts how often the cgroup hit the quota wall. throttled_usec shows how long it spent blocked there.

This is the concrete node-level form of a Kubernetes CPU limit. If a pod limit is 500 millicores, somewhere down the path a cgroup ends up with a quota roughly equivalent to half a CPU. When application teams say "the container was slow even though the node was idle", cpu.stat is often where the argument stops being philosophical.

Weights and quotas can interact badly

People sometimes treat cpu.weight and cpu.max as if one will cancel the other. They do not. The cgroup can first be favoured under contention and then still be throttled by its own quota.

That leads to a common anti-pattern:

• a latency-sensitive API is given a high weight so it wins locally
• the same API is given a tight quota to enforce tenancy
• short bursts now slam into cpu.max, producing tail-latency spikes

The scheduler is doing exactly what you asked. The policy is just internally inconsistent for bursty work.

Memory Control Is Really Reclaim Policy Plus a Local OOM Boundary

If CPU control teaches you to think in budgets, memory control teaches you to think in pressure domains.

In cgroup v2, the most important files are:

• memory.current: current charged usage
• memory.low: best-effort protection from reclaim
• memory.min: stronger protection from reclaim
• memory.high: threshold that triggers aggressive reclaim and throttling
• memory.max: hard limit that can trigger memcg OOM
• memory.stat: detailed breakdown of where memory went
• memory.events: counters for low, high, max, oom, and oom_kill

A lot of confusion disappears once you separate these knobs by function.

memory.low and memory.min are protection signals

These settings are about protecting a workload from reclaim pressure caused by neighbours.

If a cgroup is below memory.low, reclaim tries hard to take pages from elsewhere first. memory.min is stronger. It expresses a protected amount that reclaim should treat as unavailable except in more extreme situations.

These are not promises of guaranteed RAM in a hypervisor sense. They are reclaim priorities inside the host's memory-management logic.

memory.high is a pressure threshold, not a kill limit

memory.high is probably the most misunderstood memcg control.

Crossing it does not immediately kill the workload. It tells the kernel to start reclaiming more aggressively from that cgroup and to throttle tasks that keep pushing the group over the line.

That means a service can feel slow and pressured long before it dies.

If an API server has:

echo 768M | sudo tee /sys/fs/cgroup/workloads/api/memory.high

and its working set keeps growing past 768 MiB, tasks allocating memory in that cgroup may spend more time in reclaim paths. Latency climbs. memory.events shows high increments. PSI for memory gets uglier. Yet no OOM kill has happened.

Operators often miss this because they look only for death. But the real user-visible failure mode may be reclaim-induced slowdown.

memory.max defines the hard wall

memory.max is the hard ceiling.

echo 1G | sudo tee /sys/fs/cgroup/workloads/api/memory.max

If the cgroup cannot reclaim enough memory to stay within this limit while satisfying new charges, the kernel triggers an OOM decision inside that cgroup.

That last clause matters. This is not necessarily a host-wide OOM. It is a local failure domain.

This is why a container can die while the node still has plenty of memory overall. The relevant arena is not the entire machine. It is the cgroup.

A typical event file might then look like:

low 0
high 421
max 37
oom 3
oom_kill 1

And the kernel log often contains a memcg-specific line similar to:

oom-kill: constraint=CONSTRAINT_MEMCG, task=python3, pid=18422, oom_memcg=/system.slice/api.service

The exact wording varies by kernel version, but the presence of CONSTRAINT_MEMCG or an explicit cgroup path is the key clue. The kernel is telling you this was a cgroup-scoped OOM, not a global one.

Memory accounting is broader than many people think

memory.current is not just anonymous heap.

Depending on workload behaviour, the cgroup may be charged for anonymous pages, page cache, kernel memory, socket buffers, shared memory, transparent huge pages, and more. memory.stat is where that breakdown becomes visible.

A real investigation often starts here:

cat /sys/fs/cgroup/workloads/api/memory.current
cat /sys/fs/cgroup/workloads/api/memory.events
cat /sys/fs/cgroup/workloads/api/memory.stat
cat /sys/fs/cgroup/workloads/api/memory.pressure

If memory.stat shows huge file usage, the complaint may really be page cache growth. If anon dominates, you are dealing with application allocations. If sock or kernel looks large, the memory is not where the application team expected.

This is also why "the process RSS looked fine" is often a weak argument. RSS is a process view. Memcg policy applies to the cgroup's total charged footprint.

IO, PID, cpuset, and Administrative Controls Complete the Picture

CPU and memory get most of the attention, but they are not the whole story.

pids.max stops fork storms at the service boundary

The PID controller is one of the most practical safety rails in Linux.

echo 128 | sudo tee /sys/fs/cgroup/workloads/api/pids.max

This does not limit only classic processes in the everyday sense. It limits task creation through fork and clone, so heavy thread creation counts too.

When the cgroup reaches the limit, new task creation fails, commonly with EAGAIN. The existing tasks stay alive. The kernel does not kill the group just because it hit pids.max.

That behaviour is useful because it turns a runaway fan-out bug into a contained admission failure instead of a node-wide disaster.

io.weight and io.max shape block-device competition

The IO controller works at the block layer, so it cares about the underlying device, not about friendly application labels.

io.weight is the proportional-share knob. io.max is the absolute cap knob. A typical limit might look like:

8:0 rbps=10485760 wbps=5242880

That says the device with major:minor 8:0 may read at up to 10 MiB/s and write at up to 5 MiB/s for this cgroup.

The details get more device-specific than CPU or memory. Queueing model, filesystem behaviour, and the block stack all matter. The safe mental model is that cgroups can shape device competition, but the observability is less intuitive than with cpu.stat or memory.events.

cpuset is about placement, not timeslice fairness

The cpuset controller decides where tasks are allowed to run and which NUMA nodes they may allocate memory from.

The main files are:

• cpuset.cpus
• cpuset.mems

This controller solves a different problem from cpu.max. Quota answers how much CPU time the workload may consume. Cpuset answers which CPUs and memory nodes it may use at all.

That distinction matters for performance work. A database pinned to specific CPUs and NUMA nodes may have excellent locality and terrible throughput if you made the set too narrow. Another workload may have no quota cap and still perform badly because cpuset placement destroyed cache locality or forced memory access across NUMA boundaries.

A common operational mistake is to set cpuset.cpus and forget that cpuset.mems must also be sensible. CPU placement and memory-node placement are linked.

cgroup.freeze and cgroup.kill are administrative tools

Not every useful cgroup file is a resource controller.

cgroup.freeze lets you suspend execution in a whole subtree. cgroup.kill lets you terminate all tasks in that subtree in one operation.

Those are administrative primitives. They matter to service managers, checkpoint-restore tooling, and cleanup flows. They are part of why cgroups are best understood as workload control planes, not just limit files.

systemd and Kubernetes Are Both cgroup Front Ends

One of the easiest ways to make cgroups feel less abstract is to notice how often you already use them indirectly.

systemd maps units onto the hierarchy

systemd organises processes into three main cgroup concepts:

• slices for structural grouping
• services for long-lived daemons
• scopes for externally created groups of tasks

A typical host might expose paths like:

/system.slice/sshd.service
/system.slice/postgresql.service
/user.slice/user-1000.slice/session-3.scope

That is not metaphorical structure. Those are real cgroup paths.

When you set unit properties such as:

• CPUWeight=
• CPUQuota=
• MemoryHigh=
• MemoryMax=
• TasksMax=

systemd translates them into the corresponding cgroup files.

That translation is the reason systemctl set-property api.service MemoryHigh=768M MemoryMax=1G has immediate kernel consequences without you ever touching /sys/fs/cgroup by hand.

The debugging tools reflect this mapping too:

systemd-cgls
systemd-cgtop
systemctl status api.service

If you are diagnosing host services, these are often more natural entry points than navigating the raw filesystem first.

Kubernetes eventually lands in the same tree

Containers make the same journey with more layers in front.

A Kubernetes Pod spec contains resource requests and possibly limits. The scheduler reasons from requests. The kubelet receives the scheduled pod, asks the runtime to realise it, and arranges the cgroup configuration on the node. Depending on runtime and cgroup driver, the resulting path shape varies, but the underlying mechanism is still the same Linux cgroup tree.

Conceptually, the stack often looks like this:

kubepods.slice
└── burstable.slice
    └── pod-<uid>.slice
        ├── container-a.scope
        └── container-b.scope

The names differ by driver and runtime, but the pattern does not. Pod-level and container-level boundaries become cgroups. CPU limits become cpu.max. Memory limits become memory.max. PID limits and cpusets can also appear depending on policy.

This is why good node debugging ignores marketing vocabulary quickly. If a pod is slow, eventually you want the actual files and counters at the cgroup path the kubelet created.

One host can have several layers of cgroup policy

This part catches many people.

A container may be limited by:

• its own leaf cgroup
• the pod cgroup above it
• a system slice above that
• host-level policy from a platform team

The effective result is the intersection of all of these.

That is why a local file may look permissive while the workload still behaves constrained. The parent is tighter.

Delegation, Threaded Mode, and Why Some Writes Fail

Once you go beyond the common leaf-level use cases, cgroups expose a few rules that are deeply sensible and initially annoying.

Delegation is real, but bounded

Cgroup v2 was designed so a parent can safely hand a subtree to another manager.

The parent controls which controllers are delegated through cgroup.subtree_control. It can also control ownership of the subtree in the ordinary filesystem-permission sense. The delegate may then create child cgroups and tune only the controllers it was granted.

What the delegate cannot do is escape ancestor policy. If the parent capped memory at 2 GiB, the child manager cannot conjure 4 GiB by writing a larger number below.

This makes delegation useful for rootless container engines, nested service managers, or batch systems. It also means debugging nested managers requires you to ask who owns the subtree and which controllers were actually delegated.

Threaded mode exists because not every resource is naturally per-process-group

Most day-to-day cgroup work stays in domain mode, where a cgroup is a resource domain for complete processes.

But Linux also supports threaded cgroup mode for cases where threads within one process need more granular CPU control. The relevant files are cgroup.type and cgroup.threads.

You do not need this for ordinary service management. But it explains why some advanced schedulers or runtimes talk about threaded roots and why certain write patterns that seem harmless fail with mode-related errors.

The rough intuition is simple:

• CPU scheduling can sensibly distinguish threads in more granular ways
• memory accounting generally wants a coherent process-domain view

Trying to mix these carelessly produces errors because the kernel is protecting invariants, not being arbitrary.

Common write failures usually point to one of four mistakes

When a cgroup write fails, the cause is often one of these:

1. Controller not enabled in the parent
2. No internal process rule violated
3. Child asks for something outside ancestor policy
4. Threaded versus domain mode mismatch

That short list is worth memorising because it prevents a lot of random thrashing through documentation.

Observability Starts With the Right Files

The biggest cgroup debugging mistake is looking only at the application and not at the cgroup counters that explain the kernel's decisions.

A compact checklist covers most real incidents.

First locate the cgroup path

cat /proc/$PID/cgroup

Or, if systemd owns the task:

systemctl status <unit>
systemd-cgls

You need the actual path before any other number means much.

Then read the controller files that match the symptom

For CPU complaints:

cat /sys/fs/cgroup/<path>/cpu.max
cat /sys/fs/cgroup/<path>/cpu.weight
cat /sys/fs/cgroup/<path>/cpu.stat
cat /sys/fs/cgroup/<path>/cpu.pressure

For memory complaints:

cat /sys/fs/cgroup/<path>/memory.current
cat /sys/fs/cgroup/<path>/memory.high
cat /sys/fs/cgroup/<path>/memory.max
cat /sys/fs/cgroup/<path>/memory.events
cat /sys/fs/cgroup/<path>/memory.stat
cat /sys/fs/cgroup/<path>/memory.pressure

For PID explosions:

cat /sys/fs/cgroup/<path>/pids.current
cat /sys/fs/cgroup/<path>/pids.max

For placement questions:

cat /sys/fs/cgroup/<path>/cpuset.cpus
cat /sys/fs/cgroup/<path>/cpuset.mems

And always remember to inspect the ancestors if the numbers do not add up. The leaf may not be the tightest point in the tree.

Pressure Stall Information is often the missing clue

PSI files such as cpu.pressure, memory.pressure, and io.pressure are especially useful because they reveal time spent stalled, not just totals.

A service can remain alive while spending a painful fraction of time waiting on reclaim or CPU availability. Traditional utilisation graphs often smooth that pain away. PSI does not.

If memory.events keeps increasing at high and memory.pressure shows sustained stalls, you are looking at a cgroup under reclaim stress even if no OOM kill has happened. That distinction changes the fix. You may need a higher memory.high, a smaller cache footprint, or a different workload mix, not just a bigger memory.max.

The Failure Modes That Matter in Production

By the time most teams notice cgroups, they are already in trouble. The useful move is to connect the symptom to the controller.

Idle node, slow service

Classic cause: cpu.max throttling.

The node can have spare cores and the service can still be blocked because its cgroup burned through the current period's budget. cpu.stat exposes this quickly.

No OOM kill, but latency keeps climbing

Classic cause: memory.high reclaim pressure.

The workload is not dead. It is being slowed by repeated reclaim and charge throttling inside its memcg.

Process count looks normal, yet forks fail

Classic cause: pids.max counted threads or short-lived helper tasks you were not tracking.

The question is not "how many top-level processes do I see". The question is "how many tasks has this cgroup created".

Application says it uses 600 MiB, cgroup says 1.1 GiB

Classic cause: page cache, socket buffers, shared memory, or other charged memory outside the application's own simplified view.

memory.stat settles the argument.

One service changed, another slowed down

Classic cause: they share an ancestor with tight CPU or memory policy, or one sibling's weight and pressure changed the relative outcome under contention.

The tree is the unit of reasoning. Looking only at the leaf that screams is often not enough.

CPU pinning fixed jitter and broke throughput

Classic cause: cpuset solved locality but shrank the placement set too hard.

Quota, weight, affinity, and NUMA placement are distinct axes. Fixing one can worsen another.

cgroups Make More Sense Once You Stop Treating Them as Container Magic

The clean mental model is this.

A cgroup is a node in a kernel-managed tree. Tasks live in leaves or near-leaves. Parents decide which controllers children may use. Controllers expose files whose values directly shape scheduling, reclaim, IO competition, placement, and task creation. Descendants inherit ancestor reality. Service managers and container platforms are mostly nicer front ends for writing those files and organising that tree.

Once you think that way, a lot of operational mysteries stop being mysterious.

A Kubernetes CPU limit is cpu.max somewhere in the pod subtree. A systemd MemoryHigh= setting is memory.high on the unit's cgroup. A local OOM inside a container is memcg policy doing exactly what it was designed to do. A fork storm stopped by pids.max is not a random kernel quirk. It is a service boundary holding.

That is what cgroups actually are: not branding, not magic, and not merely container trivia. They are the Linux workload-control layer that turns one machine into many resource domains without pretending those domains are separate kernels.