How to Troubleshoot a Failed systemd Service on Linux
Diagnose failed Linux services with systemctl and journalctl. Read the right logs, inspect unit settings, distinguish reload from restart, and verify recovery before closing an incident.
By Network Nuts Team · Published 2026-10-14
A failed service is a symptom. Before restarting it repeatedly, collect the evidence that explains why it stopped. A short, repeatable investigation makes it easier to repair the right problem and explain the change afterward.
This guide is for Linux machines that use systemd, including RHEL. You need a terminal and permission to inspect system logs; changing services normally requires administrator privileges. Replace example.service with the actual unit name throughout. Perform changes to important services during an agreed maintenance window.
1. Capture the failure
Start with a focused status check:
systemctl --failed --type=service
systemctl status example.service --no-pager -l
Record the active state, failure result, timestamp and process exit information. An enabled service is configured for activation through its installation links; it is not necessarily running. Conversely, a running service need not be enabled.
The status screen is a starting point. A generic exit code usually needs an application log to explain it. See the upstream systemctl documentation.
2. Read the relevant journal
Inspect the service's messages from the current boot:
sudo journalctl -u example.service -b -n 100 --no-pager
Here, -u selects the unit, -b selects the current boot, and -n 100 limits output to recent entries. If the fault occurred earlier, use an appropriate time window:
sudo journalctl -u example.service --since "30 minutes ago" --no-pager
Read around the first failure, including the lines before it. A final “failed” message may only report the consequence of an earlier configuration error.
For a failure before a reboot, inspect available boots first:
sudo journalctl --list-boots
sudo journalctl -u example.service -b -1 --no-pager
Previous-boot logs must still exist for that last command to help. Some applications also write their own log files, so an empty journal is not proof that nothing happened. The upstream journalctl documentation explains these filters.
3. Compare the unit with the environment
Inspect the unit and its drop-ins:
systemctl cat example.service
A drop-in can override the main unit configuration. Read all displayed sections before deciding which setting is responsible. This behavior is documented in systemd.unit.
Check the executable path, service account, working directory and configured environment files. A command that works in your login shell may run with a different account or environment as a service.
Two useful clues from systemd.exec are:
| Clue | What to investigate |
|---|---|
203/EXEC | systemd could not execute the program. Check the executable path and accessibility. |
200/CHDIR | systemd could not enter the configured working directory. Check that directory and access to it. |
For example, if a deployment moves an application but leaves its old WorkingDirectory= in place, restarting repeatedly will not correct that path. Compare the unit setting with the deployment's intended location, make a targeted correction, then verify it.
Avoid broad permission changes as a guessing strategy. Preserve the evidence and determine which account needs access to which resource.
4. Apply the appropriate reload or restart
These operations have different purposes:
| Operation | Purpose |
|---|---|
systemctl daemon-reload | Make systemd reread unit configuration. |
systemctl reload example.service | Ask a service that supports reloading to reread its own configuration. |
systemctl restart example.service | Stop and start the service; this can interrupt requests. |
After manually changing a unit file or drop-in, reload systemd's configuration. Apply the service change using the application's documented procedure. A daemon reload does not itself restart the application. The distinction is explained in the upstream systemctl manual source.
5. Handle repeated-start failures carefully
Start-rate limiting can block further attempts when a unit starts too frequently. Diagnose and correct the underlying fault first. Then, if that limit is the remaining obstacle:
sudo systemctl reset-failed example.service
sudo systemctl start example.service
Resetting the state and counters does not repair the application. Keep rate limiting in place rather than masking a crash loop. See unit start-rate limits.
6. Verify the service's actual job
Check status and recent logs again, then test the function users depend on: a web request, a queue task, or another application-specific operation. A running process alone does not prove that its dependencies or responses are healthy.
For your next lab incident, write down four things: the symptom, the evidence, the exact correction, and the verification result. That record is more useful than a history of repeated restarts.
To develop these skills through structured Linux administration practice, explore Network Nuts RHCSA training.