How systemd Actually Works

You can use Linux for years without building a stable mental model of systemd.

You learn a few practical incantations. systemctl restart fixes a stuck daemon. journalctl -u shows logs. daemon-reload is required after editing a unit file, though people are often hazy about why. Boot time is something systemd-analyze measures. A service that used to start fine under a shell suddenly fails under PID 1 because the environment, working directory, capabilities, mount namespace, or readiness model is different.

That surface knowledge is useful, but it is not enough when a machine in Berlin hangs at boot, a timer in Amsterdam does not fire, a daemon in Athens says it is ready before it really is, or a service in Prague keeps restarting even though the main process appears healthy for a few seconds.

systemd stops feeling random once you stop treating it as a bag of commands and start treating it as a state manager.

That state manager keeps a database of units, relationships between them, queued jobs, runtime properties, and process groups backed by cgroups. It reads declarative unit files, merges them with drop-ins and generated fragments, translates start or stop requests into transactions, spawns processes under explicit execution policy, tracks their lifecycle, and records the outcome in the journal and on the D-Bus API that systemctl talks to.

That is the real subject here.

This article walks through systemd in the order that matters operationally: what it is, why Linux needed it, how units are loaded, how dependency transactions work, what really happens when a service starts, why activation is broader than boot, how cgroups and sandboxing fit in, what journald and D-Bus expose, and which failure modes actually matter when you are on the hook for a broken host.

What systemd Is

systemd is a long-running userspace manager that models a Linux system as units, dependency edges, queued jobs, and cgroup-tracked processes, then drives the machine toward a requested state.

That definition matters because it is more precise than saying "systemd is init".

Yes, on most Linux distributions in 2026, systemd is PID 1 in the system instance. It is the first long-lived userspace process after the kernel hands control to the real root filesystem. But reducing systemd to "the thing that starts services" hides most of the mechanism.

At runtime, systemd is doing at least five distinct jobs:

1. loading configuration from unit files, drop-ins, generators, and transient requests
2. solving dependencies and ordering between units
3. supervising processes and tracking state transitions
4. applying execution policy such as cgroup limits, namespaces, credentials, and filesystem protections
5. exposing state through D-Bus, systemctl, and journald

That is why systemd spans so many command families:

• systemctl for units and jobs
• journalctl for logs
• systemd-analyze for boot and dependency timing
• systemd-run for transient units
• systemd-cgls and systemd-cgtop for cgroup-backed runtime view

It is also why systemd has both a system manager and optional user managers. The system instance is PID 1. User instances run as per-user managers under user@<uid>.service, with their own units, targets, timers, and sockets. The same model appears at both levels.

One more distinction matters early: systemd is not just a collection of shell wrappers over process creation. systemctl is usually a D-Bus client talking to the manager. The source of truth is the manager's in-memory state, not the text of a unit file on disk and not a PID file left behind by a daemon.

That is the first mental shift.

Why Linux Needed More Than Boot Scripts

Classic SysV init scripts were good enough when Linux systems were simpler and expectations were lower.

A service script could implement start, stop, and restart. The boot process could walk through numbered symlinks in /etc/rc*.d/. If a daemon behaved conventionally enough, a shell script plus a PID file often worked.

The problem is that large modern machines do not behave like a neat row of independent daemons.

A real host has:

• services that can start in parallel
• sockets that should accept connections before the worker daemon exists
• mounts that appear only after devices show up
• timers that replace cron-like periodic jobs
• user sessions that create their own resource domains
• network services that are not truly ready when the parent process merely forked
• containers and scopes whose processes were not born directly from PID 1
• explicit CPU, memory, and task limits that belong to a service boundary, not to one PID

The old script model had recurring blind spots.

Serial order was too crude

A numbered startup sequence confuses two different questions:

1. what must exist before something can run
2. what may run in parallel once those conditions hold

systemd separates those questions. After= and Before= describe ordering. Wants= and Requires= describe dependency. Those are related, but they are not the same thing.

Readiness was vague

A daemon could fork into the background and write a PID file, yet still be unusable because it had not bound the socket, finished recovery, or loaded its keys.

systemd introduced explicit readiness models through Type= and later sd_notify(). That gave the manager a better answer than "the parent shell script returned zero".

Process tracking by PID file was fragile

Forking daemons, wrapper scripts, helper processes, and crash loops make PID files unreliable.

systemd tracks units with cgroups, main PID metadata, and service state. That is a much stronger model than hoping /run/foo.pid points at the process you think it does.

Activation opportunities were being wasted

Why keep a rarely used daemon running all day if the kernel can hold a listening socket and systemd can start the daemon only when traffic arrives? Why poll from cron when a timer unit can integrate with service state? Why fake dependency on a file path with shell loops when path and mount units can model it directly?

systemd's answer was not just "boot faster". The deeper answer was "model more of the operating system directly".

Unit Files Are Inputs, Not Runtime State

The next thing to understand is that a unit file is not the unit. It is one input from which the runtime unit object is constructed.

That sounds subtle. It is actually operationally important.

A system manager usually builds unit configuration from several layers:

• vendor units under /usr/lib/systemd/system/ or /lib/systemd/system/
• local overrides under /etc/systemd/system/
• runtime units under /run/systemd/system/
• drop-ins such as /etc/systemd/system/api.service.d/override.conf
• generated units under /run/systemd/generator/
• transient units created over D-Bus or by systemd-run

The merge rules matter.

If the package ships athens-api.service under /usr/lib/systemd/system/ and you create a drop-in under /etc/systemd/system/athens-api.service.d/limits.conf, the resulting live configuration is the merge of those layers. Editing the vendor file directly is usually the wrong move because package upgrades may overwrite it.

Useful inspection commands expose this merged view:

systemctl cat athens-api.service
systemctl show athens-api.service
systemd-delta

• systemctl cat shows the base file plus drop-ins
• systemctl show shows resolved runtime properties
• systemd-delta shows where local configuration diverges from vendor defaults

A representative service file might look like this:

[Unit]
Description=Athens API
Wants=network-online.target
After=network-online.target postgresql.service
Requires=postgresql.service
 
[Service]
Type=notify
User=api
Group=api
WorkingDirectory=/srv/athens-api
ExecStart=/srv/athens-api/bin/server
Restart=on-failure
RestartSec=2s
MemoryHigh=768M
MemoryMax=1G
TasksMax=256
 
[Install]
WantedBy=multi-user.target

A few details hide inside this small file.

• [Unit] describes relationships and metadata
• [Service] describes execution and supervision
• [Install] does not affect runtime dependency solving directly; it tells systemctl enable what symlinks to create

That last point trips people up constantly. WantedBy=multi-user.target does not itself start the service. Enabling the unit creates symlinks so that multi-user.target will later want it.

This also explains why changing a unit file requires:

sudo systemctl daemon-reload

The manager has already parsed unit files into in-memory objects. Editing a file on disk does not mutate PID 1 by magic. daemon-reload tells the manager to rescan unit definitions and rebuild the relevant state.

If you want static checking before you break a host, use:

systemd-analyze verify /etc/systemd/system/athens-api.service

That catches a surprising number of syntax and reference problems before they become a failed boot or a failed deploy.

Enablement, Masking, Conditions, and Assertions Are Different Levers

Another source of confusion is that systemd has several ways to make a unit start or not start, and they solve different problems.

Enablement answers “should this unit be pulled in by a target?”

systemctl enable athens-api.service usually creates symlinks under a target such as multi-user.target.wants/. That affects future activation through the install graph. It does not start the unit immediately, and it does not force the unit active right now.

systemctl disable removes those symlinks. The unit file may still exist and still be startable manually.

This is why operators sometimes get surprised by “disabled but active” or “enabled but inactive”.

• disabled but active means the service is running now, but it will not be pulled in automatically next boot or next target activation
• enabled but inactive means the install links exist, but nobody has started the unit in the current session yet

Masking is stronger than disabling

systemctl mask links the unit name to /dev/null, making start attempts fail because the unit is intentionally blocked.

That is a different tool from disable.

• disable says “do not pull this in automatically”
• mask says “this unit must not be started at all under this name”

Masking matters for incident containment and for replacing vendor behaviour safely. If a distribution ships a helper you truly do not want to run, masking is the clear statement.

Conditions and asserts are runtime gates

Unit files can also include conditions such as:

[Unit]
ConditionPathExists=/srv/athens-api/config.yml
ConditionPathIsReadWrite=/srv
AssertPathExists=/srv/athens-api/bin/server

These are not dependency edges and not install-state flags. They are runtime checks evaluated by the manager when the unit is about to run.

The distinction between conditions and asserts matters:

• a condition that fails causes the unit to be skipped cleanly
• an assert that fails causes the start to fail hard

That gives you two different operational semantics. A condition is good for optional behaviour. An assert is good when running without the prerequisite would be actively wrong.

This is far better than burying the same logic inside shell glue in ExecStartPre= because the manager now understands why the unit did or did not run.

Static units are not broken units

Some units have no [Install] section at all. Those are often called static units. They are still perfectly valid. They are simply meant to be activated by dependency, socket, timer, path, or another unit rather than by enable.

That matters because people sometimes chase “missing enable support” as if it were an error. Often it is simply the wrong activation model.

Units Form a Graph, Not a Start List

systemd manages many unit types, not just services.

Two relation families are worth separating.

Requirement relations say whether another unit should come along

Ordering relations say in which order jobs happen

The clean rule is this:

• After= does not pull anything in by itself
• Wants= does not imply a strict order by itself

That is why this is valid and common:

Wants=network-online.target
After=network-online.target

One line says "please bring it in". The other says "if it is being brought in, I run later".

If you remember only one systemd dependency fact, make it that one. A huge number of broken unit files come from treating After= as if it meant Requires= or Wants=.

A Start Request Becomes a Transaction

When you run:

systemctl start athens-api.service

systemd does not simply fork() and exec() the binary immediately.

It first builds a transaction.

A transaction is the manager's proposed set of jobs needed to satisfy the request while respecting relationships, current state, and conflicts. Jobs may include starting dependency units, stopping conflicting units, and ordering everything into a coherent plan.

If athens-api.service requires PostgreSQL and must run after network-online.target, the transaction may include jobs such as:

• start postgresql.service
• start or verify network-online.target
• start athens-api.service

The manager then checks whether the transaction is valid.

Typical reasons a transaction is rejected or altered include:

• an ordering cycle
• a conflict with already queued work
• a requirement on a masked or missing unit
• a dependency that would be pulled in but cannot be started under the requested job mode

This is why job modes exist. replace, fail, isolate, and related modes tell systemd what to do when new work collides with existing work.

You can observe the dependency graph and ordering with tools such as:

systemctl list-dependencies athens-api.service
systemctl show -p Wants -p Requires -p After -p Before athens-api.service
systemd-analyze critical-chain

critical-chain is especially useful at boot because it shows what actually sat on the longest path to the reached target. A slow service is not always a boot blocker. A fast service on the critical chain can matter more than a slow service nobody waited on.

The important point is that systemd thinks in graph transactions, not shell script sequence.

That is why it can parallelise aggressively while still keeping hard dependencies correct.

Starting a Service Means More Than Running ExecStart

Most production confusion sits here.

A service unit is not defined by one command line. It is defined by a lifecycle model.

When systemd starts a service, it may perform several phases:

1. resolve conditions and asserts
2. prepare the execution environment
3. create or enter the unit cgroup
4. run ExecCondition= if configured
5. run ExecStartPre= helpers
6. launch ExecStart=
7. interpret readiness according to Type=
8. possibly run ExecStartPost=
9. monitor the service until stop, failure, reload, or restart

That is already much richer than "run this command".

Type= decides what "started" means

This field is one of the most important in systemd.service(5).

If a database takes eight seconds to recover its write-ahead log but the unit uses Type=simple, dependent services may race ahead because systemd treats the service as started too early.

If a modern daemon still backgrounds itself but the unit uses Type=simple, the manager may follow the wrong process. If a service uses Type=notify but never sends READY=1, the unit sits in activating until timeout.

This is why service authorship is not cosmetic. The manager is only as accurate as the lifecycle you declare.

systemd tracks services with cgroups, not just PID files

Under systemd, each service normally gets its own cgroup path. That matters because the service boundary becomes visible to the kernel.

The manager can then:

• identify all processes belonging to the unit
• kill the whole service cleanly on stop
• apply resource controls such as MemoryMax= or CPUWeight=
• collect service-wide accounting instead of guessing from one PID

For a modern long-running daemon, this is far stronger than the old "write a PID file and hope that is the right process" model.

You can inspect the live state with:

systemctl status athens-api.service
systemctl show athens-api.service -p MainPID -p ControlPID -p ActiveState -p SubState -p Result
systemd-cgls /system.slice/athens-api.service

Those fields tell different truths.

• ActiveState is the broad state such as active or failed
• SubState is the finer grain such as running, dead, or start-pre
• Result explains why the last transition ended the way it did
• MainPID and ControlPID show the main process and any active control process

Restarts are policy, not shell glue

The restart model is explicit.

[Service]
Restart=on-failure
RestartSec=2s
StartLimitIntervalSec=30s
StartLimitBurst=5

This means the manager, not some external shell loop, decides when to requeue the job. That is important because the manager also knows whether the exit was clean, whether the service ever became ready, and whether the restart rate has crossed a configured limit.

A service that crashes every second is not "trying its best". It is entering a restart policy state machine.

You see that in practice as log lines such as:

systemd[1]: athens-api.service: Scheduled restart job, restart counter is at 4.
systemd[1]: athens-api.service: Start request repeated too quickly.
systemd[1]: athens-api.service: Failed with result 'exit-code'.

Those lines are the manager describing its own policy decisions.

Activation Is Broader Than Boot

A common beginner mistake is to think systemd only matters during boot.

The real model is activation.

A unit becomes active because some event, dependency, or explicit request says it should.

Socket activation

With socket activation, the socket unit owns the listening socket and the service starts only when work arrives.

A minimal pair looks like this:

# /etc/systemd/system/rotterdam-cache.socket
[Unit]
Description=Rotterdam cache listener
 
[Socket]
ListenStream=127.0.0.1:11211
 
[Install]
WantedBy=sockets.target

# /etc/systemd/system/rotterdam-cache.service
[Unit]
Description=Rotterdam cache worker
 
[Service]
Type=notify
ExecStart=/usr/local/bin/rotterdam-cache
NonBlocking=true

When the first client connects, systemd can spawn the service and pass the already-open file descriptor. The daemon reads the inherited descriptor through the usual sd_listen_fds() path.

Why this matters:

• the kernel queues connections before the daemon is fully live
• the service can be started on demand
• restarts can happen without rebinding the public socket
• dependencies become more accurate than a custom wrapper script

Socket activation is not a historical curiosity. It is a clean answer to a real operating-system problem.

Timer activation

Timer units replace a surprising number of cron patterns.

[Unit]
Description=Nightly catalogue vacuum
 
[Timer]
OnCalendar=*-*-* 02:15:00
Persistent=true
 
[Install]
WantedBy=timers.target

The paired service does the work. Persistent=true means missed runs while the machine was powered off are handled on next activation.

That is already richer than a plain cron line, because the execution becomes a normal unit with normal logs, dependencies, resource controls, and restart policy.

Path, mount, and device activation

Some work should happen when a path changes, when a mount becomes available, or when a udev-backed device appears.

systemd models those directly too. That reduces the need for shell loops that poll for files or block boot while waiting for state they do not understand precisely.

The wider lesson is that systemd treats services as one unit type inside a larger event-driven manager.

Resource Control and Sandboxing Live at the Unit Boundary

systemd became much more useful once Linux gained a strong cgroup v2 model and mature namespace features.

Every service, scope, and slice maps to the cgroup tree. That means unit-level resource policy is not advisory prose. It is concrete kernel policy.

A service like this:

[Service]
CPUWeight=200
CPUQuota=50%
MemoryHigh=768M
MemoryMax=1G
TasksMax=256

turns into live cgroup control files under the unit path. systemctl set-property is simply a high-level front end for that mapping.

This is why systemd is such an important bridge between Linux service management and Linux resource control. The service boundary is the resource boundary.

The same is true for many execution hardening features in systemd.exec(5) and related man pages.

Useful examples include:

[Service]
DynamicUser=yes
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/var/lib/athens-api
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
SystemCallFilter=@system-service

Those lines are doing real work.

• DynamicUser= allocates an ephemeral UID and GID for the service
• ProtectSystem= remounts large filesystem areas read-only for that unit
• PrivateTmp= gives the unit its own /tmp
• CapabilityBoundingSet= and AmbientCapabilities= shape Linux capability state at launch
• SystemCallFilter= turns seccomp policy into unit configuration

This is one reason systemd can feel invasive to people who remember init scripts. It is managing far more of the execution envelope than init scripts ever did.

The right reaction is not panic. It is to notice that many of these controls were always necessary. systemd simply gathered them at the place where the service boundary is already being defined.

Stop, Reload, and Kill Behaviour Are Part of the Contract Too

systemd service management is not only about getting a process to appear. It is also about stopping, reloading, and cleaning it up predictably.

Stopping a unit means applying explicit kill policy

When you run:

systemctl stop athens-api.service

systemd does not just send one signal to one PID and hope the rest sorts itself out.

It applies unit-level stop logic:

1. run ExecStop= if configured
2. send the configured stop signal, usually SIGTERM
3. wait up to TimeoutStopSec=
4. escalate if needed, usually with SIGKILL

Because the whole cgroup belongs to the unit, stop semantics can cover helper processes too. That is one reason KillMode= exists.

Common values include:

• control-group to signal every process in the unit cgroup
• mixed to signal the main process first and then the rest
• process to signal only the main process

For most modern daemons, control-group is the safest default because it matches the service boundary systemd is already supervising.

Reload is not restart

Operators often blur these together, but systemd does not.

• restart means stop and start again
• reload means ask the running service to reload configuration in place

If the daemon supports reload, express it explicitly:

[Service]
ExecReload=/bin/kill -HUP $MAINPID

or, better, point at a real admin subcommand if the service exposes one.

This matters because systemctl reload participates in manager state, logs, and failure handling. A shell user sending kill -HUP by hand gets the signal out. A unit-level reload tells PID 1 what operation was intended and whether it succeeded.

Timeout and watchdog settings change failure semantics

Startup and shutdown timing are part of the service contract too.

If a daemon under Type=notify needs thirty seconds to recover a large local cache in Frankfurt, but TimeoutStartSec= is still ten seconds, systemd is not being unfair when it kills the unit. It is enforcing the contract you described.

The same applies to watchdog support. A service that advertises WatchdogSec= is promising periodic liveness notifications. If those stop, the manager can treat the unit as wedged even if one process is still technically alive.

That is important on real fleets. “Process exists” and “service is healthy enough to keep traffic” are not the same statement.

Clean stop design is part of service design

The old init-script world often treated shutdown as an afterthought. systemd makes that hard to get away with. If a service leaks worker processes outside its cgroup, ignores SIGTERM, or needs fragile wrapper scripts to reload safely, those flaws become visible quickly.

That is a feature. A service manager should make lifecycle mistakes obvious.

Boot Is Target Resolution, Not a Magical Phase

Because systemd is PID 1 on most general-purpose Linux systems, people often think boot is its primary purpose.

Boot matters, but the useful model is still target resolution over a graph.

default.target is usually a symlink to the target the machine should reach by default, such as multi-user.target or graphical.target.

A target is not a process. It is a synchronisation point. When the machine reaches multi-user.target, that means the set of units defining that state has been pulled in and activated as far as their own dependencies and outcomes allow.

Special targets structure the sequence:

• sysinit.target for early system initialisation
• basic.target for a usable baseline of services and sockets
• multi-user.target for non-graphical multi-user mode
• graphical.target for a graphical login stack
• rescue.target and emergency.target for recovery paths

Targets are also why boot diagnostics are better under systemd than they were under a loose collection of scripts.

Useful commands include:

systemd-analyze
systemd-analyze blame
systemd-analyze critical-chain
journalctl -b

Each answers a different question.

• systemd-analyze splits firmware, loader, kernel, and userspace time
• blame shows units that took time
• critical-chain shows what actually blocked progress toward the target
• journalctl -b gives the whole boot timeline in one place

That last point matters. Boot issues often cross boundaries between the kernel, udev, mounts, generators, and ordinary services. Having one journalled timeline is a major practical advantage.

D-Bus and Journald Expose the Live Model

systemctl feels local and textual. Underneath, systemd's runtime model is exposed over D-Bus.

That is why management tools can query or manipulate unit state without scraping human-readable output. The manager owns typed properties such as:

• load state
• active state
• substate
• dependency lists
• cgroup path
• restart counters
• start timestamps
• execution result

The practical rule is simple: if you need scriptable truth, prefer stable properties over parsing status output meant for humans.

For example:

systemctl show athens-api.service -p ActiveState -p SubState -p Result -p NRestarts -p FragmentPath -p ControlGroup

That is far better automation input than grepping coloured prose.

journald plays a similar role for logs. A journal entry is not just a plain line of text. It also carries structured fields such as:

• _SYSTEMD_UNIT
• _PID
• _UID
• _BOOT_ID
• MESSAGE
• PRIORITY
• MESSAGE_ID

So when you run:

journalctl -u athens-api.service
journalctl -b -u athens-api.service
journalctl -u athens-api.service -g timeout

you are querying indexed structured data, not just reading a rotated log file.

This matters a lot during incident response. A failed service is not only a process exit. It is a unit state transition, a set of journal entries, maybe a cgroup with counters, perhaps a restart burst, and a dependency outcome visible over D-Bus.

That combination is why debugging on a systemd machine can be much faster once you learn where the manager keeps truth.

Transient Units and Scopes Explain a Lot of Modern Linux Behaviour

Not every managed process comes from a static unit file under /etc/systemd/system.

Two runtime concepts matter here.

Transient units

A transient unit is created dynamically over D-Bus rather than from a permanent file.

systemd-run is the familiar operator front end:

systemd-run --unit=backup-check --property=MemoryMax=512M /usr/local/bin/check-backup

That command asks the manager to create a transient service unit, place the process in the right cgroup, apply the declared properties, and supervise it like any other service.

This is much cleaner than starting a background process by hand and then losing track of its logs, resource policy, and state.

Scopes

A scope is systemd's way of tracking a process group created externally.

Login sessions are the most common example. A desktop session under session-3.scope was not started the same way as a classic daemon, but systemd still tracks it as a unit so it can own the cgroup, lifecycle, and accounting.

Once you know this, a lot of host behaviour becomes less mysterious.

• container runtimes can place workloads in scopes
• desktop sessions appear as scopes
• systemd-run --scope lets you attach existing command trees to managed resource policy

The point is that systemd is not only a boot-time daemon launcher. It is a live process-group manager for the machine.

The Failure Modes That Matter in Production

Most teams only care deeply about systemd when something failed. The useful move is to connect symptoms to the part of the model that actually owns them.

After= was set, but the dependency never started

Classic cause: ordering without requirement.

A unit with After=postgresql.service does not pull PostgreSQL in by itself. If you meant "start it and then run later", you probably wanted Wants= or Requires= as well.

The service becomes active before it is usable

Classic cause: wrong Type=.

A daemon that needs to warm caches, recover data, or establish listeners should often use Type=notify and send READY=1 only when it is genuinely ready. Type=simple is fine only when early activation semantics are actually correct.

The binary was edited, but nothing changed

Classic cause: no daemon-reload, or you changed the wrong layer.

Remember the load path and merge rules. The live unit object is not the text file you just saved. Check systemctl cat, systemctl show, and systemd-delta before assuming systemd ignored you.

Stop kills more processes than expected

Classic cause: the whole cgroup belongs to the unit.

This is often correct behaviour. The service boundary is the whole cgroup, not merely the main PID. If you started helper children inside the unit cgroup, stopping the unit normally stops them too.

Restarts keep happening though the program survives briefly

Classic cause: crash loop, timeout, watchdog failure, or never reaching the configured ready state.

Inspect:

systemctl status athens-api.service
systemctl show athens-api.service -p Result -p NRestarts -p TimeoutStartUSec
journalctl -u athens-api.service -b

The manager usually tells you which policy line you tripped.

A service works in a shell and fails under systemd

Classic cause: different execution environment.

systemd units may have:

• a different PATH
• a different working directory
• no interactive shell expansion
• sandboxing or capability restrictions
• different environment variables
• a different umask
• filesystem protections or private namespaces

A unit file is not a shell session. That difference is deliberate.

Boot feels slow, but the blamed unit is a red herring

Classic cause: confusing duration with critical path.

systemd-analyze blame shows which units took time. It does not prove which ones delayed the target. For that, critical-chain is the better tool.

The unit file looks right, but the start request is rejected

Classic cause: transaction conflict, masking, or ordering cycle.

At that point you are no longer debugging one process. You are debugging the manager's state graph.

That is normal. systemd is a graph solver as much as a process launcher.

The Mental Model That Actually Holds Up

systemd makes much more sense once you stop treating it as ceremony around fork() and start treating it as a state machine for Linux services and system resources.

The stable mental model looks like this:

1. unit files, drop-ins, generators, and transient requests produce runtime unit objects
2. units are nodes in a dependency and ordering graph
3. start and stop requests become transactions made of jobs
4. services have explicit lifecycle semantics described by Type=, timeouts, and restart policy
5. each service lives in a cgroup, which makes process tracking and resource control precise
6. activation can come from boot targets, sockets, timers, paths, devices, or explicit operator requests
7. journald and D-Bus expose the live result of all of the above

Once you think that way, many systemd problems stop looking mystical.

A socket-activated daemon is not magic. PID 1 is holding the listener and passing the file descriptor when work arrives. A MemoryMax= setting is not abstract configuration. It is cgroup policy on the unit boundary. A service that is forever activating under Type=notify is not being stubborn. It never sent the readiness signal the manager was waiting for. A boot that hangs on network-online.target is not haunted. Some unit in the critical chain asked the machine to wait for a network state that never became true.

That is what systemd actually is: not a religion, not a conspiracy, and not merely a replacement for init scripts. It is Linux's userspace state manager for units, process groups, activation events, and dependency-driven machine state.

The paired lab for this topic walks through that mechanism visually: unit loading, trigger selection, transaction building, cgroup-backed service start, readiness signalling, and restart policy. The quiz then checks whether you can tell requirement edges from ordering edges, interpret Type=, and recognise which part of the manager owns a given failure. If you can do those two things confidently, systemd stops being folklore and starts being predictable.